safe_json_stream v1.0.22

Safe JSON stream

This stream module differs from others in that it prevents DOS attacks by checking on the fly JSON elements and their subelements size. By doing this we avoid following:

a) out of memory problem b) blocking for too long problem

This module is intended for streaming of array, which can contain other arrays, strings, numbers....

example of valid data:

["aa", "b", {"test": "n1","n2", 1000}, "c", {"test2":"xxxxxxxxxxx"}, "d", "e", "f", "g"]

elements are checked against size in next packets:

"aa" // length of this element is 6 bytes "b" {"test": "n1","n2", 1000} // length of this element and subelements together is 27 bytes "c" {"test2":"xxxxxxxxxxx"} ...

Note: If we specify element_size_limit parameter as 20 bytes, then the third element will cause the error to be thrown from stream

Donate

My income consists of donations for my projects. If this module is useful to you, consider making a donation!

You can donate using PayPal or Wire transfer.

Note

This module is being frequently updated and newly found errors/bugs will be fixed as soon they are discovered.

Discussion

Out of memory happens in JSON streaming, when particular element in JSON array is too large. With this module this never happens, since we can adjust max number of bytes, that are contained in element (and it's subelements). With overflow, error is being thrown out of stream and we can catch it.

Blocking for too long happens with some of existing modules in following situation: You have like thousands of elements in JSON array, that are small strings. Then we change in the middle of array one string to contain a lot of characters (like 10 MB of size). While running those streams, blocking happens for some time, when this very large element is being converted from pure string to JSON dictionary. Again, this does not happen with my module.

Example

function perform_stream()
{
  return new Promise(function (resolve, reject)
  {
    var readFile = fs.createReadStream(__dirname + '/test.json', 
    {
      flags: 'r',
      'bufferSize': 11  * 1024,
	  'highWaterMark': 11 * 1024,
      'objectMode': false,
    });
	
    readFile.setEncoding('utf8');
	
    var safe_json_stream_instance = new safe_json_stream({ element_size_limit: 28 });  // limit element and it's subcontent size to 28 bytes
	 
    readFile.pipe(safe_json_stream_instance);
    readFile.on('end', function() { console.log('Reading in readFile stream is over!'); } );
	 
    safe_json_stream_instance.on('error', function(error)
    {
      reject(error);	 
    });
	 
    safe_json_stream_instance.on('data', function(income_data) // data comes in form of JSON dictionary
    { 
      console.log('Element passed : ' + JSON.stringify(income_data)); // if we need string form of incoming data, we use JSON.stringify
    });

    safe_json_stream_instance.on('end', function() { console.log('JSON safe stream is over!'); resolve(); });
  });
}