@gannochenko/protoweb NPM

About the Project
Getting Started
Usage
Commands
Troubleshooting
- Optional and Mandatory Fields
Best Practices
Gotchas
Roadmap
Development
Contributing
License
Built With
Contact

About The Project

Protoweb is a tool for converting proto definitions into arbitrary JS/TS code that could be used in your organization.

Protocol buffers is an industry standard when it comes to building direct service-to-service communication. However, when used with the Google annotations extension to enable REST on-top of gRPC, it quickly becomes a hassle when running the thing in a front-end project, such as React or Vue-based one.

Engineers face three options:

Write and maintain service definitions manually, or
Use AI, such as Chat-GPT for one-time conversions, or
Use end-to-end solutions, such as Connect.build.

Definitions written manually can be an option, but as the project grows, it always becomes more and more difficult to support the written files and track new changes.

AI is certainly an option too, but not easily applicable when running inside CI/CD pipelines (even though it is possible).

While Connect.build is a great tool, it's also the "all-or-nothing" solution, which means one can't use it partially, they should commit to it fully. It turns into a problem in case when an organisation has, say, a custom implementation of fetch or an SDK that entirely wraps the networking API up in order to enable transparent JWT refreshing or a retry mechanism.

This is where Protoweb comes to the rescue: it converts messages to TS type definitions using protoc and protoc-gen-ts, and then flavours it with service definitions of your choice. These services can later be used in parts of the front-end app, for instance, in connection with react-query.

Write your proto files once, then never write typescript manually.

Getting Started

Prerequisites

Node
Yarn or NPM (comes with Node)
Protoc
ts-proto

Installation

yarn global add @gannochenko/protoweb

Installation to a different folder

To avoid problems with permissions and sudo, it is possible to install the package locally and then add it's folder to PATH variable.

Create ~/.node folder
```
mkdir ~/.node
```

Install the package

yarn global add @gannochenko/protoweb --prefix ~/.node

Add ~/.node folder to PATH
```
export PATH=${PATH}:${HOME}/.node/bin
```
Add the command above to ~/.bashrc (or ~/.bash_profile in case of MacOS)
You should be able to run the protoweb CLI command now

Upgrading

If you followed the way how the installation was done, then do upgrading as following:

yarn global upgrade @gannochenko/protoweb --prefix ~/.node

Usage

The generator needs three things:

Full path to the folder where the protobuf definitions are kept.
The template for a service file.
The output path.

Example

Let's say there is a protobuf definition stored in ~/someservice/protobuf:

syntax = "proto3";

package someservice.image.v1;

option go_package = "backend/proto/v1/imagepb";

import "google/api/annotations/annotations.proto";
import "common/image/v1/image.proto";
import "common/page_navigation/v1/page_navigation.proto";

message ListImagesRequest {
  common.page_navigation.v1.PageNavigationRequest page_navigation = 1;
}

message ListImagesResponse {
  string version = 1;
  repeated common.image.v1.Image images = 2;
  common.page_navigation.v1.PageNavigationResponse page_navigation = 3;
}

service ImageService {
  rpc ListImages(ListImagesRequest) returns (ListImagesResponse) {
    option (google.api.http) = {
      post : "/v1/image/list"
      body : "*"
    };
  }
}

There is also a template ~/cool_service_template.cjs written in JavaScript (not TypeScript!) and using CommonJS (not ESModules!):

const ejs = require('ejs');

const template = `import { ErrorResponse, apiUrl } from "../../../util/networking";

<%- protocOutput %>
<% services[0].methods.forEach(method => { %>
// <%= method.comment %>
export async function <%= method.name %>(request: <%= method.requestType %>): Promise<<%= method.responseType %> | ErrorResponse> {
  try {
    const response = await fetch(\`$\{apiUrl\}<%= method.url %>\`, {
      method: "<%= method.verb %>",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(request),
    });

    if (!response.ok) {
      return {
        error: \`Status: $\{response.status\}\`,
      };
    }

    return await response.json();
  } catch (error) {
    return {
      error: (error as Error).message,
    };
  }
}
<% }); %>
`;

module.exports = {
    renderTemplate: (data) => {
        return ejs.render(template, data);
    },
};

We can then compile the definitions to ~/my-output:

mkdir ~/my-output
protoweb build -i ~/someservice/protobuf -o ~/my-output -t ~/cool_service_template.cjs

The result will become:

import { ErrorResponse, apiUrl } from "../../../util/networking";

// Code generated by protoc-gen-ts_proto. DO NOT EDIT.
// versions:
//   protoc-gen-ts_proto  v2.6.1
//   protoc               v3.19.4
// source: image/v1/image.proto

/* eslint-disable */
import { type CreateImage, type Image } from "../../common/image/v1/image";
import {
  type PageNavigationRequest,
  type PageNavigationResponse,
} from "../../common/page_navigation/v1/page_navigation";

export const protobufPackage = "someservice.image.v1";

export interface ListImagesRequest {
  pageNavigation: PageNavigationRequest | undefined;
}

export interface ListImagesResponse {
  version: string;
  images: Image[];
  pageNavigation: PageNavigationResponse | undefined;
}

export interface ImageService {
  /** ListImages returns a list of user images, paginated and sorted by creation date */
  ListImages(request: ListImagesRequest): Promise<ListImagesResponse>;
}

// ListImages returns a list of user images, paginated and sorted by creation date
export async function ListImages(request: ListImagesRequest): Promise<ListImagesResponse | ErrorResponse> {
  try {
    const response = await fetch(`${apiUrl}/v1/image/list`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(request),
    });

    if (!response.ok) {
      return {
        error: `Status: ${response.status}`,
      };
    }

    return await response.json();
  } catch (error) {
    return {
      error: (error as Error).message,
    };
  }
}

Now use this file with react-hooks, easy.

Templating mechanism

Protoweb uses ejs as a template engine for its internal template. You are free to use anything else in your custom template, but whatever you use should be resolvable relative to the location of your template file. It means there ought to be the node_modules folder next to it, or in a one of the parent folders, and npm install or yarn install must be executed beforehand.

The custom template should export the renderTemplate() function that takes data as an argument and returns a string.

The structure of data is the following:

{
    protocOutput: string; // output produced by protoc, contains type definitions
    services: { // services parsed from the protobuf file
        name: string; // service name
        methods: { // methods parsed from the protobuf file
            name: string; // method name
            requestType: string; // name of the request type
            responseType: string; // name of the response type
            url: string; // url parsed from the google annotations
            verb: string; // HTTP verb parsed from the google annotations
            comment: string; // optional comment
        }[];
    }[];
    ejs: typeof ejs; // an instance of ejs, so there is no need to import it in a template
    sourcePath: string; // path to the source proto file
    destinationPath: string; // path to the destination typescript file
    convertSnakeToCamel: (input: string) => string; // a helper that converts snake to camel
    processURLPlaceholders: (url: string, prefix: string) => string; // a helper that converts and escapes placeholders in URLs, such as /{foo}/{bar}/{baz}
    ucFirst: (input: string) => string; // a helper that uppercases the first letter
    ast: object; // a deeply nested structure that contains an AST of the corresponding protofile (just in case if you need it, which I doubt to be honest)
}

Note, that protocOutput must be present in the template file, and should not be escaped.

Json Decoder

Protoweb supports generation of JsonDecoders. While TypeScript gives static type safety, JsonDecoers implement runtime safety: if an actual response from the server comes malformed, an exception will be thrown.

Basically, what Protoweb does is - for every message Foo it creates a complimentary Decoder, that mirrors the structure of that message. Example:

message ProductResponse {
  string sku = 1;
  int32 quantity = 2;
}

will be converted to:

export const ProductResponseDecoder = JsonDecoder.object(
    {
        sku: JsonDecoder.optional(JsonDecoder.string),
        quantity: JsonDecoder.optional(JsonDecoder.number),
    },
    "ProductResponse"
);

Features supported:

regular messages,
nested messages (but please don't use them),
oneOf,
enums.

Decoders will be attached to the value of the protocOutput variable. Import statements will be modified in order to import decoders that come from other files. JsonDecoder will also be imported.

If messages are not declared in the right order in the protobuf file, decoders will be, because otherwise TypeScript throws an error that a variable was used before initialized.

Commands

👉 Specify -r only when you wish to compile only part of the proto files in a project. The builder still needs to know the root folder, because there could be imports inside.

Example:

protoweb build -i ~/coolcompany/protobuf/coolproject -r ~/coolcompany/protobuf/ -o ~/my-output -t ~/cool_project_service_template.cjs

Type protoweb -h to find out about all available commands.

Troubleshooting

Q: I see errors such as "Error parsing file XYZ: Error: illegal name '}'".

A: Look into the failing protobuf files, check if you have empty annotations such as option (google.api.http) = {}. Either remove them or fill it in, otherwise the protobuf parser Protoweb uses considers this illegal syntax 😕.

Q: Why do all my 64-based numbers get converted to string all of a sudden?

A: Protoweb converts all 64-based numbers to string, because it's unsafe to convert them to number, and you probably don't want any extra library for managing long numbers in your bundle. Use 32-based numbers instead.

Optional and mandatory fields

By default, all fields in Json Decoders will have JsonDecoder.optional() wrapper. This is reasonable, because Protobuf syntax3 (which we all should follow) clearly states, that all fields are optional. However, this leads to use of bothersome optional chaining operator (?.) of JavaScript. Also, the decoders will be not strict enough.

If you trust your server that it populates all the fields in the responses, you can use the --with-json-decoder-required-fields option. Protoweb then will do its best to declare fields as non-optional. However, there are exceptions:

field that is a part of the oneOf statement is always optional,
fields that are annotated with @protoweb: optional comment in the protobuf will be rendered as optional,
dates will be still nullable, no matter what (since they are being transferred as strings, and that string can be an invalid ISO date).

Best practices

Avoid declaring recursive messages. Protoweb can't generate decoders in such cases.
Avoid nested messages (even though they are supported, it's a mess).
Avoid 64-based int64, uint64, float64, etc. They will be converted to type String, because they cannot be represented as type Number in JavaScript. You will have to use BigInt or some LongAr library otherwise. Just avoid these types, this is not what you want.

Gotchas

Normally you only need decoders for responses, because you don't trust your own fellow server. However, currently Protoweb generates decoders for all messages it was able to find. This happens because otherwise we would need to analyze the declaration of services first, then make a dependency tree and shake off all un-involved messages. Then build decoders on the rest. This functionality is 100% feasible to build, it's just not there yet. Please use tree shaking on your end, otherwise some unwanted stuff may sneak into your bundle.

When specifying --with-json-decoder-required-fields all basic fields in types will still be mandatory and object fields will have | undefined modifier. Currently I can't do much about it, but you can use the following type modifier in your template:

type DeepNonUndefined<T> = T extends object
     ? {
         [K in keyof T]: DeepNonUndefined<Exclude<T[K], undefined>>;
       }
     : T;
 
 type DeepReplaceDateWithNullable<T> = T extends Date
     ? Date | null // Replace Date with Date | null
     : T extends object // Check if it's an object or array
         ? {
             [K in keyof T]: DeepReplaceDateWithNullable<T[K]>;
           }
         : T;

And then do return data as DeepNonUndefined<DeepReplaceDateWithNullable<ProductResponse>>.

Roadmap

Bugfixing :)
Support for additional tooling such as JsonDecoder. This is a debatable feature, because usually Backend we run is a trusted entity. However, TS definitions only provide static checks, we may want to have some runtime checks in place as well in case we make requests to third-party services. This is a doable task, yet a tricky one: the Protobuf AST must be compiled into AST of JsonDecoder, and then to a schema of JsonDecoder. There could be a lot of edge cases to cover down the road.

See the open issues for a list of proposed features (and known issues).

Development

Clone the repo

git clone https://github.com/gannochenko/protoweb.git

Install NPM packages
```
cd protoweb;
yarn;
```

Build and run

yarn start build -i <folder> -o <folder>;

There is a built-in template that will be used should the -t option be missing.

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.