# safe_json_stream

> Safe JSON stream

Latest version **1.1.3** (published 2024-06-04) · MIT license · 0 weekly downloads

## Install

```sh
npm install safe_json_stream
pnpm add safe_json_stream
yarn add safe_json_stream
bun add safe_json_stream
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.1.3 |
| Published | 2024-06-04 |
| First published | 2016-05-29 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 2 |
| Unpacked size | 2.1 MB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Maintainers | kgregor |

## Links

- npm: https://www.npmjs.com/package/safe_json_stream
- npm.io page: https://npm.io/package/safe_json_stream

## Dependencies (2)

- [util](https://npm.io/package/util.md) ^0.10.3
- [stream](https://npm.io/package/stream.md) 0.0.2

## Recent versions

- 1.1.3 (latest) — 2024-06-04
- 1.1.2 — 2024-06-04
- 1.1.1 — 2024-06-04
- 1.1.0 — 2024-06-04
- 1.0.22 — 2018-06-25
- 1.0.21 — 2018-06-25
- 1.0.20 — 2018-06-25
- 1.0.19 — 2016-05-31
- 1.0.18 — 2016-05-31
- 1.0.17 — 2016-05-31
- 1.0.16 — 2016-05-31
- 1.0.15 — 2016-05-31
- 1.0.14 — 2016-05-30
- 1.0.13 — 2016-05-30
- 1.0.12 — 2016-05-30
- … 19 more at https://npm.io/package/safe_json_stream/versions

## README

Safe JSON stream

-----
This stream module differs from others in that it prevents DOS attacks by checking on the fly JSON elements and their subelements size.
By doing this we avoid following:

a) out of memory problem<br>
b) blocking for too long problem



This module is intended for streaming of array, which can contain other arrays, strings, numbers....

example of valid data:

[["aa"], ["b"], {"test": ["n1","n2", 1000]}, ["c"], {"test2":"xxxxxxxxxxx"}, ["d"], ["e"], ["f"], ["g"]]

elements are checked against size in next packets:

["aa"]                           // length of this element is 6 bytes<br>
["b"]<br>
{"test": ["n1","n2", 1000]}     // length of this element and subelements together is 27 bytes <br>
["c"]<br>
{"test2":"xxxxxxxxxxx"}<br>
...

Note: If we specify element_size_limit parameter as 20 bytes, then the third element will cause the error to be thrown from stream


Donate
-----

My income consists of donations for my projects. If this module is useful to you, consider <a href="http://kgregor.si/other/donate.html">making a donation</a>!

You can donate using PayPal or Wire transfer.


Note
------
This module is being frequently updated and newly found errors/bugs will be fixed as soon they are discovered.
This version differs from previous, in that JSON string elements are returned from stream. This is for reasons of speed.
If you need JSON dictionary, just use JSON.parse on returned strings.
I am planning in near future to create even faster version of Safe JSON stream, which would be implemented as Node.js C++ addon.


Discussion
-----

Out of memory happens in JSON streaming, when particular element in JSON array is too large. With this module this never happens,
since we can adjust max number of bytes, that are contained in element (and it's subelements). With overflow,
error is being thrown out of stream and we can catch it.

Blocking for too long happens with some of existing modules in following situation:
You have like thousands of elements in JSON array, that are small strings. Then we change in the middle of array
one string to contain a lot of characters (like 10 MB of size). While running those streams, blocking happens
for some time, when this very large element is being converted from pure string to JSON dictionary.
Again, this does not happen with my module.


Example
-------

```
async function perform_stream()
{
  var readFile = fs.createReadStream(__dirname + '/test.json', 
  {
    flags: 'r',
	'bufferSize': 11  * 1024,
	'highWaterMark': 11 * 1024,
	'objectMode': false,
  });
   
  readFile.setEncoding('utf8');
	
  var safe_json_stream_instance = new safe_json_stream({ element_size_limit: 8 });  // limit element and it's subcontent size to 8 bytes

  safe_json_stream_instance.on('data', function(income_data) // data comes in form of string variable
  { 
    // if we need JSON dictionary form of incoming data, we use JSON.parse
    console.log('Element passed : ' + income_data); 
  });
  
  try
  {
    await pipeline(
    readFile,
    safe_json_stream_instance
   );
  }
  catch (e)
  {
    console.log('Error: ' + e.toString());
	  return;
  }

  console.log('JSON safe stream is over!');
}

```
<br>
Full source code of example is included along module.

---
_Source: https://npm.io/package/safe_json_stream · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._