Codis-master NPM

CODIS - CODIS-MASTER

CODIS - A system for distributed computing using JS engines in web browsers.

CODIS-MASTER - The system's server part.

Manages executive nodes (slaves).
Stores information about tasks.
Stores information about executive nodes.
Runs the CODIS task system script written by the system's user.

CODIS-MASTER action also needs CODIS-SLAVE package, which must be used in the service held with the website on which we want to create worker threads.

CODIS-SLAVE-SCRIPT

Optionally, we can also start the service with the administration panel of our system (CODIS-ADMINISTRATION-PANEL). It does not have to work in the same environment as CODIS-MASTER.

CODIS-ADMINISTRATION-PANEL

Installation

$ yarn global add codis-master

$ npm install -g codis-master

Getting Started

After installing the package, you must call the command:

$ codis-init-master

The command created the system configuration file codis.config.js and a directory /plan will be created and the system's initial script of the system tasks: /plan.js.

To start the system, you must call the command:

$ codis-run-master

Configuration

The default configuration file contains:

exports.config = {
    port: 5299,
    acceptedOrigins: ['http://localhost:3000'],
    threadNumberPerNode: 4,
    planLocalization: '\\plan\\plan.js',
    administrationPanel: {
        publishAPI: true,
        port: 9251,
        refreshRateNodesPage: 5000,
        refreshRateTasksPage: 5000,
        refreshRateSystemPage: 1000,
    },
};

###Options:

port - the port on which the system is to be exposed
acceptedOrigins - all acceptable origins from which communication to the system will come. It means that only on these given origins the system will allow to run threads of executive nodes. There can be many possible values. In order for the system to accept each origin, you set true in the array: acceptedOrigins: [true],
threadNumberPerNode - determine how many threads will be run at a single execution node. That is, if we have the value set to (for example 4), then when opening a page with an injected CODIS-SLAVE script, browser will run 4 working threads under the given tab in the browser / or the entire browser. The larger the number ~ the greater efficiency of a single execution node (of course, only to some extent).
planLocalization - location of the initial system script
administrationPanel
- publishAPI - providing api for the administration panel CODIS-ADMINISTRATION-PANEL
- port - the port on which the API is to be exposed
- refreshRateNodesPage - frequency of refreshing nodes information ms
- refreshRateTasksPage - frequency of refreshing tasks information ms
- refreshRateSystemPage - frequency of refreshing general system information ms

Plan of system tasks

The initial content of the file plan.js

exports.run = (CODIS) => {};

In the body of the function, we write the whole logic of what the system has to do. The CODIS object that we receive as a parameter of this function is used for communication with the system's memory. The CODIS object provides two methods to use:

createTask()
controlNotification()

createTask()

Function definition:

createTask: 
(module: string, task: any, options?: TaskOptions, parentTaskId?: number) 
=> {promise: Promise<any>, promiseResolve: (PromiseLike<T> | T)) => void, promiseReject: (reason?: any) => void) => void, idTask: number, createSubTask: createTask}

Parameters:

Only two fields are required to provide: task, module. The other arguments are optional.

module - the name of the module to be downloaded for the given task. The definition of a given module created by the user allows to perform appropriate calculations for a given set of input data of tasks (given as the second function argument). Otherwise it is a file with the given module name with the extension .mjs. For example: exampleModule.mjs
task - the content of the task to be handled and executed by the module specified in the first createTask() argument. Task can have any format.
parentTaskId - we give the identification number of the task, which will be the main (parent) task for the task being created.
options - definition of TaskOptions:

TaskOptions {
    endless?: boolean;

    lazyLoadingData?(task: Task): any;

    handleRegisterSolution?(task: Task, solution: any): any;
}

endless - when we want to create a task that will be performed by execution nodes endlessly. Each free system thread will be sent the specified task.

lazyLoadingData - function that will be called when a free thread comes to the task. The function should return the content of the task for the thread (that is what is normally given as the second argument of the createTask () function). This can be done so that the data that is to be used as the content of the task are not kept in the system's memory. But it also creates a real way for the data to be extracted, for example, from external sources (databases or files). The argument of the function that we can use is the content of the task originally given as the second argument of the createTask() function. And it should be remembered that the content of this original task will not be sent to the execution thread, if we have the given function lies the LoadingData in the options of the task being created (unless in this function we actually return this value).

handleRegisterSolution - a method of use similar to that offered by lazyLoadingData. Only that this function is called when we get a response from the execution thread where our tasks were processed. And this function gets just a solution to a given task and decides what to do next with a given solution (eg save to an external database or to a file). As the first argument receives the details of the task, in the second argument we get received solution.

The returned object from createTask()

You should know that the entire communication process of the script written by the user and the system relies on Promise objects from JavaScript. When creating a task, Promise is created and returned, which is connected to the memory. This promise will be solved when the task is resolved. Thanks to this, we can create an extensive and task-dependent system logic.

The returned object has 5 fields:

promise - it is an object of type Promise. It will be resolved only when the task is resolved by one of the nodes and returns the value of the solution.
promiseResolve, rejectPromise - thanks to these functions, we can solve or reject a given Promise object, which is assigned to the created object, from the level of the script related to plan.js. Give the value of the solution to the function.
idTask - identification number of the created task. We can use it at a later stage to, for example, create dependencies with other tasks or to report solutions or errors related to this task.
createSubTask - function through which we can create tasks depending on the task. The syntax and return values of this function are the same as createTask, but the difference here is the fact that the last argument parentTaskId is already filled with the identification number of the task just created.

const task = CODIS.createTask('exampleModule', {task});
const subTask = task.createSubTask('exampleModule', {task2});

is the same as:

const task = CODIS.createTask('exampleModule', {task});
const subTask = CODIS.createTask('exampleModule', {task2}, null, task.idTask);

controlNotification()

Function definition:

controlNotification: (message: string, type: NotificationType = 'info') => void;

type NotificationType = 'error' | 'primary' | 'info' | 'custom' | 'warning' | 'success';

This function is mainly used to display notifications / information on the console of the administration panel.

We can choose the category of notification from the available list of options NotificationType. Each of these types will be displayed in a special way and color.

Modules

An important part of the system are modules. So the scripts that are downloaded as threads of executive nodes. The single module is a single script with the extension .mjs. In these files we define the logic of solving incoming tasks to the thread. Modules should be provided by the service that serves the website and be stored in a location designated by CODIS-SLAVE.

For more information, go to the CODIS-SLAVE package page:

Simple Examples

The examples will be illustrated by the example of the system for finding prime numbers:

In the first example, we want to count the number of prime numbers in the range of 1 - 100,000:

exports.run = (CODIS) => {
    let promises = [];
    let counter = 0;

    CODIS.controlNotification('Create task', 'primary');
    for (let i = 1; i < 100000; i++) {
        promises.push(CODIS.createTask("primeNumber", {n: i}).promise.then(
            () => {
                CODIS.controlNotification(`Prime number found: ${i}`, 'info');
                counter++;
            }));
    }
    Promise.all(promises).then(
        () => {CODIS.controlNotification(`Found ${counter} prime numbers`, 'success');}
    );
};

In the second example we will extract numbers from the database and found prime numbers will be stored in the same database. And we will do it forever.

const getNumberFromDatabase = () => {
    const number = con.query(`CALL get_number()`);
    return {n: number};
};

const savePrimeNumber = (task, solution) => {
    if (solution.isPrimeNumber) {
        const number = solution.number;
        con.query('INSERT INTO prime_num SET ?', {number})
    }
};

exports.run = (CODIS) => {
    CODIS.createTask("primeNumber", null, {endless: true, lazyLoadingData: getNumberFromDatabase, handleRegisterSolution: savePrimeNumber});
};

In the third example, we'll create a parent thread over threads to check prime numbers. The parent's thread will accept incoming primes from the sub-tasks threads.

exports.run = (CODIS) => {

    const parentTask = CODIS.createTask("collectNumbers");
    for (let i = 1; i <= 100000; i++) {
        parentTask.createSubTask("exampleModule", {n: i});
    }

    parentTask.promise.then((solution) => {
        CODIS.controlNotification(`Prime Numbers: ${solution.primeNumbers.toString()}.`, 'success');
        CODIS.controlNotification(`Composite Numbers: ${solution.compositeNumbers.toString()}`, 'success');
    });
};

The examples presented are quite trivial but clearly illustrate the possibility of the system. In order to write complex applications, you'll need to use JavaScript knowledge.

In progress

sending data from parents to children (tasks)
sending messages from the administration panel to the selected node
sending messages from the main node to the executive nodes
full integration and functional tests (both for CODIS-Slave and CODIS-master)

Future

launching executive nodes not only in the browser but also as scripts / applications from the disk
full application save points (for action against CODIS master server failures)