jsonstream2-native v1.0.0

JSONStream2-native

This module is based on JSONStream, but uses node's streams2 rather than hacking in something else.

Most importantly, this means that this streaming JSON parser actually respects backpressure when piped.

The input side is a normal Writable stream that takes bytes; the output side is an objectMode Readable.

JSONStream2.parse(path, map, opts)

path and map are the same as JSONStream. There are three opts:

• encoding - The input (writable) stream's text encoding.
• writableHighWaterMark - The input (writable) side's highWaterMark in bytes. This is the buffer level when write() starts returning false. Default = 16 kB.
• readableHighWaterMark - The output (readable) side's highWaterMark in objects. When the readable buffer is full, JSONStream2 will stop parsing incoming bytes until parsed objects are read out. Default = 128

A note on memory usage

Parsed objects can take up significantly more memory than raw bytes sitting in a Buffer. Additionally, you want to avoid buffering in the first place by informing your source (ie the file you've piped in) to pause the deluge of bytes until downstream work is done. Therefore, it is very important that you set an appropriate highWaterMark on any streams that you pipe this in to. In objectMode, node's default of 16,384 applies to the count of objects, not the actual size of the data in those objects.

If you're dealing with objects that are large or contain large strings, you may quickly exhaust your RAM by buffering thousands of objects if one of your downstream streams is slow to consume.