# @scottsloane/pipelinejs

> A data pipeline library for Javascript

Latest version **0.1.3** (published 2024-02-28) · WTFPL license · 0 weekly downloads

## Install

```sh
npm install @scottsloane/pipelinejs
pnpm add @scottsloane/pipelinejs
yarn add @scottsloane/pipelinejs
bun add @scottsloane/pipelinejs
```

## Health

**Score 20/100 (F)** — status: abandoned.

Positive: esm support; no vulnerabilities.

Warnings: low downloads; no types; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.1.3 |
| Published | 2024-02-28 |
| First published | 2024-02-27 |
| Weekly downloads | 0 |
| License | WTFPL |
| TypeScript types | none |
| Module format | ESM + CommonJS |
| Dependencies | 0 |
| Unpacked size | 14.6 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Scott Sloane |
| Maintainers | scottsloane |
| Keywords | pipeline, data, transform, etl |

## Links

- npm: https://www.npmjs.com/package/@scottsloane/pipelinejs
- Repository: https://github.com/scottsloane/pipelinejs
- Issues: https://github.com/scottsloane/pipelinejs/issues
- npm.io page: https://npm.io/package/@scottsloane/pipelinejs

## Alternatives

- [@libsql/sqlite3](https://npm.io/package/@libsql/sqlite3.md) — 39.8K weekly downloads
- [@fortemi/core](https://npm.io/package/@fortemi/core.md) — 461 weekly downloads
- [cdb-converter](https://npm.io/package/cdb-converter.md) — 341 weekly downloads
- [@uplo/adapter-prisma](https://npm.io/package/@uplo/adapter-prisma.md) — 75 weekly downloads
- [typeorm-aios](https://npm.io/package/typeorm-aios.md) — 30 weekly downloads

## Recent versions

- 0.1.3 (latest) — 2024-02-28
- 0.1.2 — 2024-02-27

## README

# PipelineJS

PipelineJS is a data pipeline library for Javascript. It is intended to bring some of the design pattern advantages of channel based piplines from lanuages like Go to Javascript for projects that benifit from Javascripts other advantages.

> Note: the Javascript event loop limits what performance gains can be achieved with a channel based pipeline. However, the design pattern is still very helpful for certain project types.

## Installation

Install with *npm* or *yarn*

`` yarn add @scottsloane/pipelinejs ``

`` npm i --save @scottsloane/pipelinejs ``

## Usage

### Import the Pipeline class from the library

```javascript 
import Pipeline from '@scottsloane/pipelinejs' 
```

### Create you segment functions.

*See example below*

### Insance a new pipeline passing it your segment functions.

```javascript
const pipeline = new Pipeline(catalyst, [func1, func2]); 
```

### run the pipeline

Inside an async scope:

```javascript
await pipeline.run();
```

## Example 
```javascript
import Pipeline from 'pipelinejs';

const Data = [];
for (let i = 0; i < 100; i++) {
    Data.push(i);
}

function* catalist() {
    while (Data.length > 0) {
        yield Data.shift();
    }
}

function square(data) {
    console.log(`Square: ${data}`);
    return data * data;
}

function consumer (data) {
    console.log(`Consumer: ${data}`);
}

(async () => {
    const pipeline = new Pipeline(catalist, [square, consumer]);
    await pipeline.run();
    console.log('Pipeline: done');
})();

```

## The Pipeline Class

The pipeline class takes three arguments.

1. **function\***: Catalyst
2. **Array\<segment/subpipeline\>**: Segments
3. *(otional)* **function**: Terminator

> Note: If no terminator is specified, a noop terminator will be added to prevent back stalling on the channels.

The pipeline class has two public member functions:

### setContext(data)

Set context allows the addition of context data that will be made available to every segment. This is helpful for things like sharing a database connection pool across the entire pipeline.

### async run(void)

Runs the pipline, starting all segments.

## Subpipelines

A Subpipeline is a pipeline that is embeded in the middle of a parent pipeline. These subpipelines allow for parallel processing within a pipeline.

> Note: a Subpipeline can not contain a catalyst or terminator.

A Subpipeline can be added in the Pipeline constructor by adding a segment that is an array of two items:

1. **Number**: the number of parallel pipes to create
2. **Array\<segment\>**: The segments in the Subpipeline

```javascript
const pipeline = new Pipeline(catalyst, [func1, [4,[subfunc1, subfunc2]], func2]);
```

## Segment Types

### Catalyst*

Generator function with a channel output, used to start a pipeline.

### Function (Segment)

A standard segment function. Takes data from a channel and outputs a return result to another channel. 

### Function* (GeneratorSegment)

A generator segment function. Takes data from a channel and yields output to another channel. Useful if the function generates a lot of data.

### Terminator

A function segment without an output channel. Used to terminate (or end) a pipeline. (needed to prevent back stalling the channels)

---
_Source: https://npm.io/package/@scottsloane/pipelinejs · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._