# supertask-cluster

> A cluster superset of Supertask

Latest version **0.1.5-alpha** (published 2016-01-06) · MIT license · 0 weekly downloads

## Install

```sh
npm install supertask-cluster
pnpm add supertask-cluster
yarn add supertask-cluster
bun add supertask-cluster
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.1.5-alpha |
| Published | 2016-01-06 |
| First published | 2016-01-02 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 3 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 0 |
| Author | Schahriar SaffarShargh |
| Maintainers | schahriar |
| Keywords | cluster, task, queue, supertask, parallel |

## Links

- npm: https://www.npmjs.com/package/supertask-cluster
- Repository: https://github.com/schahriar/supertask-cluster
- Homepage: https://github.com/schahriar/supertask-cluster#readme
- Issues: https://github.com/schahriar/supertask-cluster/issues
- npm.io page: https://npm.io/package/supertask-cluster

## Dependencies (3)

- [async](https://npm.io/package/async.md) ^1.5.0
- [shortid](https://npm.io/package/shortid.md) ^2.2.4
- [supertask](https://npm.io/package/supertask.md) ^1.2.2

## Alternatives

- [cron](https://npm.io/package/cron.md) — 4.9M weekly downloads
- [@vercel/queue](https://npm.io/package/@vercel/queue.md) — 731.6K weekly downloads
- [create-sonicjs](https://npm.io/package/create-sonicjs.md) — 1.6K weekly downloads
- [@exellix/jobs-api](https://npm.io/package/@exellix/jobs-api.md) — 941 weekly downloads
- [@forwardimpact/libskill](https://npm.io/package/@forwardimpact/libskill.md) — 575 weekly downloads

## Recent versions

- 0.1.5-alpha (latest) — 2016-01-06
- 0.1.4-alpha — 2016-01-06
- 0.1.3-alpha — 2016-01-05
- 0.1.2-alpha — 2016-01-03
- 0.1.1-alpha — 2016-01-03
- 0.1.0-alpha — 2016-01-02

## README

# supertask-cluster
[![Build Status](https://travis-ci.org/schahriar/supertask-cluster.svg)](https://travis-ci.org/schahriar/supertask-cluster)
[![Test Coverage](https://codeclimate.com/github/schahriar/supertask-cluster/badges/coverage.svg)](https://codeclimate.com/github/schahriar/supertask-cluster/coverage)

## Create, compile and run tasks across a cluster with minimal setup.

# Installation
Note that this module is a superset of [Supertask](https://github.com/schahriar/supertask) and requires *ES6* & NodeJS 4.x+.
```javascript
npm install supertask-cluster
```

# Features
- Automatic Cluster & Worker creation
- Cluster Monitoring, Worker resurrection
- Bidirectional argument passing and results from Master to Worker and vice versa.
- Processing & Compilation of function or JS source on Workers with minimal effort. (uses NodeJS VM no interpreters or parsers)
- Buffer upload mechanism and Buffer references as arguments
- Entirely written in JS and fully based on NodeJS core modules where possible

# Usage
Create a new shared task with a unique name and distribute it to all available Workers (one per core). Note that a unique name is required for every task.
```javascript
var SupertaskCluster = require('supertask-cluster');
var cluster = new SupertaskCluster();
// Deploy Cluster across all CPU cores
cluster.deploy();

TaskManager.addShared('taskname', function power(n, x, callback) {
    // n^x function
    callback(null, Math.pow(n,x));
}, function callback(error, task) {
    // You'll need to distribute the task to Workers
    // before executing the latest code/context
    task.distribute(function(error, stats){
        console.log("Task was distributed to", stats[0], "cores of", stats[1], "total");
    });
});
```

Run task. You can pass arguments after name and before callback. **This will automatically run the task on a free Worker based on the internal load.**

```javascript
TaskManager.do('taskname', 2, 4, function callback(error, result) {
    console.log("2^4 is equal to", result);
});
```

Note that the usual [Supertask](https://github.com/schahriar/supertask) methods work as this module is merely a superset. 

## API
### Buffers
Buffers allow you to pre-upload a StorageType (currently only NodeJS's internal Buffer object is supported) to a Worker in order to save upload time. This is inspired by GPU's parallelization mechanism whereby data is uploaded to the GPU in order to save bandwidth and data transfer time that will otherwise make parallelization on GPUs slow. This same concept applies to a Cluster and if large Buffers are passed as arguments to every function every time you'll spend most of your processing time uploading data. This is where **Buffers** come in. Buffers are automatically chunked & uploaded (like a stream) to avoid congestion that would otherwise make passing of a 2GB Buffer to a Worker impossible.

#### Here is how to use Buffers:

Create a new task:
```javascript
// Create a large 20MB buffer on Worker ID 0
var buffer = new Buffer(20000000);
// Fill buffer with 'c's
buffer.fill('c');
// Create new Buffer on Worker 0. Note that the Buffer will take some time to upload.
// (approximately 20s per 1GB or 400ms for 20MB)
TaskManager.createBufferOnWorker('0', 'largeBuffer', buffer, 'utf8', false, true, function(error){
    if(error) throw error;
    // Here we create a reference to the Buffer
    var ref = cluster.workerBufferReference('largeBuffer');
    // Create a new task to process some of the Buffer
    TaskManager.addShared('bufferProcessor', function (buf, callback) {
        if(!buf) return callback(new Error('Buffer did not exist in the cluster'));
        // Convert 10 chars to String starting from 2kb (results in 10 'c's in a row)
        var str = buf.toString('utf8', 2000, 2010);
        // Pass str to master
        callback(null, str);
    }, function (error, task) {
        // Here we set the permissions to Minimal in order to make the Buffer object available to the task
        task.permission(SupertaskCluster.ST_MINIMAL);
        // Distribute the task to all Workers
        task.distribute(function(){
            // Run task with ref as an argument
            task.call(ref, function(error, rstr){
                // Processed string
                console.log(rstr);
                // Output: cccccccccc
            });
        });
    });
});
```
Note that the Cluster automatically chooses the best candidate that has the Buffer reference available. If the Buffer reference is not available in any of the workers it will be set to `undefined`.

A Buffer can be immutable.
[Read API documentation for more info](./documentation/api.md).
More documentation and methods coming soon. Check out the test functions for more information in the mean time.

## Disclaimer
This module is not *yet* ready to be used in a production environment. While Supertask has reasonably good stability with over 40 tests it does not fully expose all methods and capabilities and may not function as intended. Supertask-cluster is equally missing some important cluster monitoring methods to keep the cluster alive and well in a production environment. Use it at your own risk.

## License
MIT © Schahriar SaffarShargh <info@schahriar.com>

---
_Source: https://npm.io/package/supertask-cluster · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._