@kavibios/node-etl v0.2.0

node-etl

Framework for extracting and transforming data using Javascript.

Structure

Jobs are defined in the folder src/rules/<jobname>/ and registered in src/index.mjs.

A job is a list of stages (which stages should be run can be controlled with the --stages parameter). Stages are always run in the order defined in the job.

A stage is a list of rules. Rules are always run in the order specified in the stage definition.

A rule is composed of 3 phases.

1. The extract phase
2. The transform phase
3. The load phase

The extract phase provides the raw data. It can be extracted from different sources - in this project it is almost exclusively extract from the naturdatabasen source. This is a specialized source extending the sql source with syntactic sugar for getting registrations from a program. The sql source simple returns an array of rows by executing the given sql against the configured database. This is often used in stage2.

The transform phase takes each row from the extract phase and applies a logical transformation to it. This can be similar to a map function, but other transformations on the input array is possible.

The load phase loads the resulting data into an output. This could be a csv or json file, but in this project it is typically an sql loader, which loads each item in the array into a row in the specified table. Note that the columns MUST be specified expicitly.

Extractors