nodepolis-aas v0.1.10

:zap:nodepolis:zap:

A sane general purpose distributed streaming computation framework in NodeJS that just works! :metal:
You may call me NodeJS streams on Steroids .

We paid :sweat::cry: - we learned, and so behold ... :eyes:
A minimalistic framework which helps to scale your data processing ambition without being invasive.
"Coz we :heart: NodeJS, Dockers & most of all you"

Nodepolis is meant to be easy to use, safe, fast, scalable and resource efficient.
It provides radical high-level API in Javascript . While imposing little to no restriction on processing operations.
And did we mention ... we are fully compatible with NodeJS streams ?

Streaming Word count example using nodepolis:

const byline = require('byline');
const fs = require('fs');
const _ = require('lodash');

//your average NodeJS stream -->
var lineStream = byline(fs.createReadStream('./sample.txt', { encoding: 'utf8' }));

//Boom - here comes nodepolis
const Context = require('nodepolis').Context;
const nc = new Context();
nc.stream(lineStream) // convert into distributed stream
  .flatMap(line => line.split(/\W/))
  .par(4) //parallelize the operation
  // we convert all functions to async for you :)
  // utilizing Nodejs power and efficiency.
  .map(_.countBy)
  // wait a minute - Did i just used lodash ???
  .byTimeWindow(5 * 1000) //every 5 seconds compute of words.
  .collect(console.log)

nc.run(); //Start my stream !!!  

If you had async computation tasks - this would still seamlessly work.
example:

flatMap(line => Promise.resolve(line.split(/\W/)))

making it great fit for tasks waiting on other processes like - network , hardware or external systems.

Features

• Distributed streams which are compatible with NodeJS streams.
• Docker optimized, Kubernetes support using Jobs & Cron.
• Zero server installation - just a docker infrastructure like Kubernetes or Swarm.
• Intelligent DAG(Directed Acrylic Graphs) workflows - support building complicated workflows with ease.
• Better work control - stop, restart or resume failed tasks. Makes it optimum for building long running workflows.
• Support for Throttling/Rate limiting streams.
• Aggregation/Stream Window support For example: compute every thirty seconds the number of cars passed in the last minute.
• Minimalistic : Low learning curve, easy to use, easy to scale. - works standalone or in distributed environment.

Docs & community

• Contributing guide
• Nodepolis User's Guide
• Roadmap

Install

Run npm install after init your npm project:

$ npm install nodepolis --save

Or install nodepolis-cli:

$ npm install -g nodepolis-cli

Then init project with nodepolis dependencies.

$ nodepolis-cli myproject

Docker

A sample docker image and compose file are provided, to run a minimal distributed nodepolis stack for experimentation purpose, on a single machine or on a swarm cluster, as explained here.

Tests

To run the test suite, first install the dependencies, then run npm test:

$ npm install
$ npm test

License