@elastic.io/batching-library v2.0.3

batching-library

Library for batching in elastic.io. Batching allows users to combine several calls to API in one big call.

BatchClient

BatchClient represents client that works with batches. It contains logic for creating, updating statuses and storing batches.

Constructor

new BatchClient(store, config)

BatchClient constructor expects three parameters: 1. store - implementation of IStore interface. For example MongoStore. This is where batches will be stored. 2. config - instance of BatchConfig. It represent configuration and defines limitation for batches e.g: maxSize, maxItemsNumber.

Example

import { MongoStore, IStore } from '@elastic.io/batching-library';
import { BatchClient } from '@elastic.io/batching-library';
import { BatchConfig } from '@elastic.io/batching-library';
import { ConnectionOptions } from 'mongoose';

const maxSize = 1000;
const maxItemsNumber = 10;
const maxWaitTime = 1000;
const maxEmitRate = 1;
const maxRetry = 0;
const uri: string = 'url_to_mongo';
const collection = 'collection_name';
const connectionOpt: ConnectionOptions = {
  user: 'user',
  pass: 'password', 
};
const mgStore: IStore = new MongoStore(collection, uri, connectionOpt);
const config = new BatchConfig(maxSize, maxItemsNumber, maxWaitTime, maxEmitRate, maxRetry);
const client = new BatchClient(mgStore, config);

Methods

1. saveItem(item)

Save provided item to batch, returns batch with saved item (without another Batch items).

BatchItem properties

Example

const batch: Batch = await client.saveItem({ id: 0, item: {}});

2. getReadyBatches()

Using provided emitter emit batches with status 'READY'

Example

const batches: Batch[] = await client.getReadyBatches();
  await Promise.all(batches.map(async (batch) => {
    try {
      
     // process batch implementation
      
      await client.updateBatchStatusById(batch.id, 'SUCCESS'); // batch was successfully processed
    } catch (e) {
      log.error('Error: %o', e);
      await client.updateBatchStatusById(batch.id, 'FAILED'); // batch processed with error
    }
  }));

Implementing your own batch storage

For reference implementation example take a look at MongoStore 1. Extend AbstractStore and override each abstract method. 2. Implementation must ensure that each operation is transactional

Environment variables

Environment template file

Limitations

MongoStore:

1. Supported Mongo database versions: 4.0 and higher.

2. Retries do not implemented.

3. Emit rate not implemented.

4. Library is not guarantee sequence processing of batch item

MaesterStore:

1. Emit rate not implemented.