outbox-nestjs v1.0.1

Transactional Outbox NestJs library

This library implements the transactional outbox pattern using Nestjs and Typeorm. For more information on transactional outbox patter, check here

Currently, the library supports event publishing to Kafka and RabbitMQ

Installation

npm install outbox-nestjs -s

Context

Outbox pattern guarantees data consistency for services that, as part of a business process, needs to persist an entity and also publish and event/message to a broker.

To achieve this, the message is stored in the database as part of the transaction that updates the business entities. A separate process then sends the messages to the message broker. This library implements this using the polling publisher pattern

The message that is persisted in the database is called the Outbox entity which looks like below

Outbox Entity

@Entity()
export class Outbox {

   @PrimaryGeneratedColumn(`uuid`)
   id: string;

   @Column()
   aggregateId: string

   @Column()
   messagePayload: string

   @Column()
   eventType: string

   @CreateDateColumn()
   createdAt: Date
}

Configuration

App Module configuration

In your NestJs application, navigate the app.module.ts and make the following configuration. The configuration can either one of the two below

• Kafka Publisher configuration
• RabbitMQ configuration

However, regardless of the event queue, there are two common configuration properties as can be seen below

Kafka Publisher Configuration

If you will be publishing to Kafka, your configuration should look something like this below. We are going to add the OutboxModule in the imports section of the@Module decorator

import {MessageQueueType, OutboxModule} from "outbox-nestjs";

imports: [
  OutboxModule.forRoot({
    kafkaOptions: {
      clientId: `test-application`,
      brokers: [`localhost:9092`],
      topic: `outbox-topic`
    },
    queueType: MessageQueueType.KAFKA,
    datasource: connectionSource
  }),
]

RabbitMQ Configuration

If you are using RabbitMQ, your configuration should look something like this below. We are going to add the OutboxModule in the imports section of the @Module decorator

import {MessageQueueType, OutboxModule} from "outbox-nestjs";

imports: [
  OutboxModule.forRoot({
    rabbitOptions: {
      exchange: `sample-exchange`,
      connectionString: `amqp://guest:guest@localhost:5672`
    },
    queueType: MessageQueueType.RABBIT_MQ,
    datasource: connectionSource
  }),
]

Datasource Configuration

This library requires a datasource in order to fetch data from the outbox table. Typically, your datasource configuration would look something like below

import { Outbox } from "outbox-nestjs";


const config: TypeOrmModuleOptions = {
   type: "xxxx",
   host: `xxx`,
   port: xxxx,
   username: `xxxx`,
   password: `xxxx`,
   database: `xxxx`,
   entities: [`xxx`, Outbox],
   migrationsRun: true,
   migrations: ["dist/domain/migrations/*{.ts,.js}"],
   autoLoadEntities: true,
   synchronize: true,
   logger: "advanced-console"
};

The important part of the above configuration is that we have to add the Outbox entity to the entities array

Polling publisher

The library uses nestjs ScheduleModule to poll and publish to the message broker. You need to initiate the module. In app.module.ts, add the following

@Module({
   imports: [
      ScheduleModule.forRoot()
   ],
})

Database Migrations

The Typeorm configuration from above has synchronize:true, which would create the Outbox table in the database, however, DO NOT USE THIS IN PRODUCTION!!!

You should create the Outbox table in your database, choosing the right data type for each property

Other Considerations

There are some details to be aware of when using this library

• The default polling interval is 1 Second for now
• This library would try to publish the events at least once. However, there can be cases where an event is published more than once.
  Your clients should be Idempotent
• While the implementation tries to publish the events in order which they arrive, the order may not be guaranteed ;)

Author

Nriagu Chidubem