Clusterizer NPM

Clusterizer

Instant clusterization of an array of module paths, a directory of modules, or an array of npm module names.

Modules don't need to be performing the same task, as is usually the case with Node.js clusters. Clusterizer excels at offloading long-running operations into another process and scheduling them intelligently.

Clusterizer includes built-in sleep-type scheduling as well as advanced scheduling through Agenda.

Installation

$ npm install clusterizer

Features

automatic process forking
events to/from each module
log/error aggregation to master process
built-in sleep-type scheduling
integration with Agenda for advanced scheduling
graceful shutdown

Usage

See test_modules/module1.coffee for an example module. Modules need to inherit from Clusterized and optionally implement a process(callback) function. The class name (or constructor function name if in js) is irrelevant as long as it is a module-level export as shown below.

{ Clusterized } = require 'clusterizer'

class Worker extends Clusterized
  process: (callback) ->
    # do something
    @send "result", 3.141592653589793
    callback(err)

module.exports = Worker

Events are another way to interact with your worker:

{ Clusterized } = require 'clusterizer'

class Worker extends Clusterized
  constructor: ->
    @on 'start-process-function', (msg) ->
      @myProcessFunction

  myProcessFunction: ->
    # do the magic
    @send "magic", "result"

module.exports = Worker

Then instantiate a Clusterizer in your code with an options object. See the example main function in clusterizer.coffee. Use .isMaster as a check to prevent your other code from running in every process.

{ Clusterizer } = require 'clusterizer'

clusterizer = new Clusterizer
  logging: true
  dir: ["../test_modules"]

if clusterizer.isMaster

  # example log handler
  clusterizer.on 'log', (msg, module) ->
    console.log "LOG : #{module} : #{msg}"

  # example error handler, err object contains fields
  # similar to Error, i.e. name, message, stack
  clusterizer.on 'error', (err, module) ->
    console.error "ERROR : #{module} : #{err.stack}"

  # example user-defined message handler
  clusterizer.on 'echo', (msg, module) ->
    console.log "\nGot #{msg} from #{module}\n"

  # modify sleep backoff time for all
  clusterizer.setSleep 500

  # modify sleep backoff for specific module
  clusterizer.setSleep 500, 'module2'

  # set agenda for all
  clusterizer.setAgenda 'localhost:27017/test', '3 seconds'

  # set agenda for one
  clusterizer.setAgenda 'localhost:27017/test', '3 seconds', 'module1'

  # start all (uses Agenda mode for any module which has an agenda set)
  clusterizer.start()

  # broadcast event to all modules
  clusterizer.send "echo", "test broadcasted message"
  # or
  clusterizer.broadcast "echo", "test broadcasted message"

  # event with message to single module
  clusterizer.send "echo", "call me back", "module2"

  # event with no message to single modules
  clusterizer.send "echo", null, "module2"

  # stops module1
  clusterizer.stop 'module1'

  # stops all
  clusterizer.stop()

  # restart all
  clusterizer.start()

  # kill all
  clusterizer.kill()

  # ... your code ...

Other forms of specifying worker modules in the Clusterizer options:

file: ["../test_modules/module1.coffee", "../test_modules/module2.coffee"]
npm: ["clusterizer-test-module1", "clusterizer-test-module1"]

Note

the file:, dir:, and npm: options can be used simultaneously
duplicate modules are currently not supported
the 'error' event is emitted, so it needs to have a listener or an unspecified error will be thrown
any errors thrown in clusterized modules will be emitted by the clusterizer with a full stack trace

Advanced Scheduling

setAgenda(database, every, name)

Use setAgenda to define a fuzzy execution frequency. Clusterizer uses Agenda behind the scenes so the database and every parameters are what Agenda expects. For example, something like

clusterizer.setAgenda 'localhost:27017/test', '3 seconds', 'module1'

If the name parameter isn't specified, Clusterizer will apply the agenda to all clusterized modules.

Calling start() once an Agenda has been defined for a module will always use the agenda, not the sleep period.

A module can call setAgenda() on itself if the db address is fixed or handed in using a message. This allows each module to specify its preferred schedule.