parali v1.4.1

Example project:

{
    "cache": "$out_dir/$project",
    "process_limit": 3,
    "ctx": {
        "out_dir": "path/to/user",
        "project": "project_name",
        // this is required for this step here
        "params": "for context",
        "repeat": ["file_1.txt", "file_2.txt", "file_3.txt"]
    },
    "params": {
        "step1": {"param": "value"}
    },
    "tasks": {
       "step1": {
           "out": ["some_var", "other_var"],
           "steps": [
            "modules.assert.checkFiles",
            "modules.assert.checkEnvironment",
            {"repeat": "modules.tasks.repeat"},
            "modules.tasks.doThis"
        ],
        "step2": {
            "in": "some_var",
            "out": ["etc", "list_of_todos"],
            "steps": "modules.filter.listOfNames"
        },
        "pre_step3": "step1",
        "step3": {
            "out": "some_other_var",
            "steps": "modules.magic.repeatedCheck"
        },
        "post_step3": "step1",
        "stepX": {
            "steps": {"list_of_todos": "module.run.for.every.todo.in.parallel"}
        }
    },
    "tasklists": {
        "default": [
            "@init",
            "step1",
            [
                "step2",
                "pre_step3",
                "step3",
                "post_step3",
                "step2"
            ],
            "stepX"
        ],
        "init": ["step1", "step2"]
    }
}

def execute(ctx):pass

module.exports = ctx => {}

when Runs in parallel:

def execute(ctx, unique_list_val): pass

Usage

npx parali path/to/project.jsonc stepX @run step1

Note If you run without parameters, then it will assume you have defined a "default" in tasklists.

Features

• Supports Python & NodeJS
• Runs tasks of a pipeline defined in jsonc like in the example above
• Can exchange data between Python and NodeJS tasks through ctx caching mechanism
• Completely seamless py & js environment
• Can run steps in parallel
• Supports concurrent processes writing to the same object all at once
• Can cache tasks for next run to have them cached if their outputs exist in cache already
• Binary Writing (type bytes) for serialisation
• base.jsonc in root directory will be the basis for all projects.
• Can reference tasks in terms of one another

Note: Each module must die with a process non-zero error code in order for the pipeline to stop.

Note: Pipeline cache is a means to an end for each step. Meaning the things that go in cache should be final and immutable so that if you do define something progressive, that must be an extended ctx cache key, otherwise cache itself won't work as there is nothing to cache when two separate steps modify the same thing.

TODO

• Compute "process_limit" value (extremely hard especially with unpredictable processes)
• Speedup cache by deeper key lookup (perhaps use nosql or the likes)
• add "local" as a jsonc configuration per task next to "in" and "out"
• Ability to run parali over multiple machines
• Progressbars of some kind, especially for parallel steps
• Header progress bar like so:
• Assert in task if "in", "out", or "local" is in any of the "params" or "ctx" and error out
• Assert in task if "in" is defined prior to "out" of a different step
• Assert in task if pipeline has multiple "out" with the same name in different steps
• Assert in task if "in" is also an "out" before running task
• Assert if "cache" can do $var replacements
• Assert if tasklist or tasks provided exist