node-concurrency v0.0.5-beta

node-concurrency

Asynchronous execution across multiple processes with Node Clusters.

How It Works

Using the mechanism intended to sync up Node Clusters, we pass serialized data between processes to allow for the execution of arbitrary user functions in worker processes rather than from the main process's Event Loop.

The bulk of the magic is achieved using simple serialization tricks; Just a bit of an augmentation on JavaScript's standard JSON functionality.

Getting Started

Installation:

npm install --save node-concurrency

Initialization:

const Master = require('node-concurrency');
const Async = new Master();

Async.startWorker();

Note that the amount of workers that can be started using Async startWorker() is unlimited, provided you start at least one. However, the general consensus is to start one per CPU, as in:

const numCPUs = require("os").cpus().length;

for (let cnt = 0; cnt < numCPUs; cnt++) {
  Async.startWorker();
}

Usage:

Async.execute(() => {
  console.log('Hello from process #' + process.pid);
});

The function called above can also take optional arguments, provided after the initial function argument. The call returns a JavaScript Promise. The value returned from the provided function is then passed along the subsequent chain.

let promise = Async.execute((parentPid) => {
  let msg = 'Hello to process #' + parentPid +
      ' from process #' + process.pid;

    return msg;
}, process.pid);

Which can, of couse, be extended by:

promise.then((msg) => {
  console.log(msg);
});

The initial function argument can also be executed after a delay, given as milliseconds after the initial function argument and before any optional extra arguments:

let promise = Async.executeTimeout(() => {
  console.log('greetings from 5 seconds in the future');
}, 5000);

The initial function argument given above will be executed, as expected, but will be delayed by 5 seconds.

External dependencies can be injected into worker processes via the master object:

// https://www.npmjs.com/package/one-liner-joke
Async.setDependency('OneLinerJoke', 'one-liner-joke');

let promise = Async.execute(() => {
  let joke = this.OneLinerJoke.getRandomJoke();

  console.log(joke.body);
});

Caveats:

Scope

Because user code executed by the execute function is to be done so from a different process, it is safe to assume that any global scope is not available to the function, as it would be within the scope it was declared.

For example, the following code will fail, due to an Unknown Identifier error on val:

let val = 'Hello';

Async.execute(() => {
  console.log(val);
});

Try something like this instead:

let val = 'Hello';

Async.execute((msg) => {
  console.log(msg);
}, val);

So we can pass values into the function instead of using the global scope. Magically, we can do this with most values, functions, objects, etc.

See, however, that each of the two examples above implies access to the console global. This is correct because it is provided by the Node runtime rather than from another module, not loaded as part of the runtime.

Consider each worker process as an independent instance of the Node interpreter. As such, you would still have access to globals like console and process but currently cannot automatically load other libraries in the worker processes. Check it out:

Async.execute(() => {
  return process.pid;
}).then((workerPID) => {
  console.log('master pid: ' + process.pid);
  console.log('worker pid: ' + workerPID);
});

The current implementation of this module only really allows for arbitrary functions, however.

Native Functions and Circular References

Much of the module's functionality is derived from it's serialization, which is capable of working with values, objects and functions alike. However, it is currently not capable of working with native functions as much of it's serialization is not new technology, but rather leveraging existing technology.

As such, this module is also unable to work with circular references.