@bstruct/gcp-pubsub-to-bigquery-example-transformer v0.0.12

gcp-pubsub-to-bigquery-example-transformer

Example how to customize the pubsub message transformation step in component gcp-pubsub-to-bigquery

One of the exported classes must implement the interface IPubsubMessageTransformer so that the transformation can be implemented inside the transform method.

The class must contain a public constructor with no parameters expected.

Here's the example in the file exampleTransformerSync:

import { IPubsubMessage, IPubsubMessageTransformer } from '@bstruct/pubsub-message-types';
import { MyNewMessageType } from './types/myNewMessageType';
import { MessageBody } from './types/messageBody';

export class ExampleTransformerSync implements IPubsubMessageTransformer<MyNewMessageType> {

    transform(message: IPubsubMessage): MyNewMessageType[] {

        const body = JSON.parse(message.data.toString()) as MessageBody;

        if (!body.orderId) {
            throw Error("Mandatory property 'OrderId' is defined in the message body.");
        }

        const newMessage = {

            orderId: body.orderId,
            status: body.status,
            date: body.date,

            __messageId: message.messageId

        } as MyNewMessageType;

        return [newMessage];
    }

}

The method can be syncronous or asynchronous, as demonstrated in the file exampleTransformerAsync.ts.

Deployment

This code is meant to be deployed to a npm package registry. In the file github-artifacts-publish.yml is how this package is deployed to github's package registry.

name: Publish package

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: 16
          registry-url: https://npm.pkg.github.com/ 
          scope: '@bstruct'
      - run: npm install
      - run: tsc
      - run: jq "del(.devDependencies)" package.json > package2.json ## do not install development packages
      - run: rm package.json
      - run: mv package2.json package.json 
      - run: cp package.json ./dist/src
      - run: cp LICENSE ./dist/src
      - run: cp README.md ./dist/src
      - run: npm ci
      - run: npm publish ./dist/src/ --access public ## only publish the code files, not the test ones
        env:
          NODE_AUTH_TOKEN: ${{secrets.GITHUB_TOKEN}}