Mongo-oplog-reader NPM

mongo-oplog-reader

Read MongoDB oplogs from a sharded cluster.

Supports multiple processes for high availability
Emits event after document has been written to the majority of members of the shard's replica set
Resumes from the last emitted event of the replica set (in the case of process crash/stop/restart)
Ability to add/remove shards/hosts without restarting

Notes:

The order of the oplog events is not guaranteed to be strictly chronological if there are multiple workers per oplog

Requirements

Redis 3.2+

Install

npm i mongo-oplog-reader redis

Usage

import MongoOplogReader from 'mongo-oplog-reader';
import redis from 'redis';

const redisClient = redis.createClient();

const connectionStrings = [
  'mongodb://shard0-primary/local',
  'mongodb://shard0-secondary0/local',
  'mongodb://shard0-secondary1/local',
  'mongodb://shard1-primary/local',
  'mongodb://shard1-secondary0/local',
  'mongodb://shard1-secondary1/local'
];

const reader = new MongoOplogReader({
  redisClient,
  workersPerOplog: 1 // total # of redundant workers per oplog (respected across all processes)
});
reader.filter(oplogDocument => oplogDocument.op !== 'u'); // ignore 'update' operations
reader.setConnectionStrings(connectionStrings);
reader.onEvent(data => {
  // return a promise to apply backpressure on the oplog stream to prevent a 
  // build-up of in-memory stream buffering while performing slower async operations
  return somethingAsync(data);
});
reader.start();