Turbo-json.js NPM

turbo-json.js is a tool that combines all json files found in a directory into one big json file using streaming to avoid out-of-memory.

npx turbo-json.js data/
# Outputs `combined.json` file containing the content of all json files found in the data/ directory

All json files found in data directory will be combined into one file that is named combined.json par default.

Both read and write actions are done using streaming. The maximum data stored in memory is the buffer size.

Installation

CLI:

Global install to have the CLI accessible from everywhere on your operating system.

npm install -g turbo-json.js # install globaly

No installation needed when using npx.

npx turbo-json.js [options]

Library:

# yarn
yarn add turbo-json.js

# npm
npm install turbo-json.js

Usage

turbo-json.js takes one argument:

Input directory: the path to the directory containing all the jsons.

and options:

Output file optional : path to the output file (default: combined.json).
Validate Input JSON's: All input JSON files are validated to be a correct JSON (default: true).
Validate Output JSON's: Validate if the outputed JSON is a correct JSON (default: false).
Quiet mode : no logs are outputed (default: false).

It accepts relative path but also fully qualified paths.

CLI usage:

Usage:

turbo-json [options] <input-directory>

Options:

  -o, --output-file <file-name>      File name in which all the json files will be merged (default: "combined.json")
  -I, --validate-input <file-name>   Check if input JSON files are valid (default: true)
  -O, --validate-output <file-name>  Check if output JSON is a valid JSON (default: false)
  -q, --quiet                        Quiet mode, no logs are outputed (default: false)

Example

turbo-json /data -o combined_data.json

Library usage

const { combineJson } = require('./src');

(async () => {
  await combineJson('misc', { outputFile: 'combined_data.json', validateInput: true, validateOutput: false, quiet: false });
})();

How is it combined

The tool requires the path to a directory to work. Inside that directory all json files are read and merged into one big json file.

The JSON file outputed by this tool contains an array with all the inputed JSON files.

Example:

If your JSON file contained { "id": 1 } it will be stored in the output file like this [{"id": 1}]

Some examples:

Input files:

{ "id": 1 }

{ "id": 2 }

Output file:

[
  { "id": 1 },
  { "id": 2 }
]

Array exception: There is one exception to this rule. If your JSON file contains an array, it will be deconstructed/flattened in the final file (could become an option please make an issue if you'd like that).

Input files:

[ 1, 2, 3 ]

{ "id": 1 }

Output file:

[
  1,
  2,
  3,
  { "id": 1 }
]

Validate Json Format

By default the tool supposes that your JSON files are correctly formated. If you'd like to check if it is well formated before concatenation the validateInput options should be set to true. By doing so, only the well formated JSON's are concatenated but this comes at the price that every file will be stream-read two times. Once for validation and once for writing.

The same is possible to validate the output file using validateOutput.

Validity is based on the description provided by json.org.