@karmaniverous/batch-process v0.1.0

batch-process

In the real world, most async operations against large data sets need to be batched and throttled.

The batchProcess function provides a simple, configurable utility for batching and throttling async operations.

batchProcess logs the progress of batch operations to console or to an injected logging dependency.

API Documentation • CHANGELOG

Installation

npm i @karmaniverous/batch-process

Usage

import { batchProcess } from '@karmaniverous/batch-process';

type Item = Record<string, unknown>; // Your data type.

// Say you have a function processBatch that processes a batch of items, for
// example writing them to a database. Maybe not every item is processed
// successfully, so the function returns a count of processed items and an
// array of unprocessed ones.
interface AsyncResult {
  processed: number;
  unprocessed: Item[];
}

const processBatch = async (items: Item[]): AsyncResult =>
  doSomethingAsync(items);

// Let's write a function that extracts any unprocessed items from an
// AsyncResult.
const unprocessedItemExtractor = ({ unprocessed }: AsyncResult) => unprocessed;

// Here's a bunch of items to process.
const items: Item[] = [
  ... // Your items here.
];

// Now you can asynchronously process a bunch of items in throttled sets of
// parallel batches!

// batchProcess will...
// - break items into batches of no more than `batchSize` items, and
// - process up to `throttle` batches in parallel, and
// - write a debug log message for each batch attempted, and
// - wait `delayIncrement` ms with exponential backoff to retry failed batches, and
// - throw an exception after `maxRetries` failed retries, and
// - return an array of AsyncResults generated during batch processing.
const results = await batchProcess(items, {
  processBatch,
  unprocessedItemExtractor,
  // see other options below
});

batchProcess Options

The options parameter has the following properties:

Default Batchable options are:

Built for you with ❤️ on Bali! Find more great tools & templates on my GitHub Profile.