2.0.1 • Published 3 years ago

yieldable-json v2.0.1

Weekly downloads
2,945
License
Apache-2.0
Repository
github
Last release
3 years ago

yieldable-json

This library provides asynchronous version of standard JSON.parse and JSON.stringify APIs.

Use Case

Node.js based web applications dealing with large JSON objects while requiring high level of concurrency for the transactions.

Here 'largeness' applies to any of, or all of, or any combination thereof deep compositions, large number of fields, and fields with large data (such as massive strings).

In contrast to JSON Streams, which are built with custom stream oriented protocols (at library as well as application level) and used in special use cases like data analytics, monitors, etc., yieldable-json works on fully formed JSON data that is characterized only by object size and designed with an objective of improving concurrency in the application and thereby usable in any workload scenarios.


Statistics

Testing with a wide range of data showed that objects which are bigger than 256KB (when stringified) degrades concurrency in the application.

While definition of concurrency is subjective, a number of tests we performed in short and long round-trip networks showed that being available in the event loop for attending to concurrent requests at every 5 milliseconds provides the required responsiveness to clients, as well as meets the overall concurrency expectation on the application.

As shown in the table, the yieldable-json guarantees the event loop is reached and processed in every 5 ms, irrespective of the JSON volume being processed.


Background

In Cloud based deployments, we foresee increase of JSON payload across distributed computing end-points in an exponential manner, causing performance bottleneck in the single-threaded node.js platform, and hence this attempt. The key function of asynchronous JSON APIs are to perform marshalling and un-marshalling of massive data incrementally, and yielding back to the application occasionally, and reduce loop starvation.

The ES6's generator function along with yield semantics is used for implementing incremental traversal. The extent of traversal in one iteration is controlled by the intensity parameter. While there are some globally accessible variables within main APIs, predominently the parser and stringifier is implemented through a self-recursing loop. The callback is guaranteed to fire no earlier than next tick to emulate the asynchronous behavior.

Because of the usage of ES6 semantics, this module is supported only on node v4.x and above.


APIs

stringifyAsync:

stringifyAsync(value, replacer, intensity, callback)

  1. If you return a String, that string is used as the property's value when adding it to the JSON string.
  2. If you return a Boolean, "true" or "false" is used as the property's value, as appropriate, when adding it to the JSON string.
  3. If you return any other object, the object is recursively stringified into the JSON string, calling the replacer function on each property, unless the object is a function, in which case nothing is added to the JSON string.
  4. If you return undefined, the property is not included (i.e., filtered out) in the output JSON string.
const yj = require('yieldable-json')
yj.stringifyAsync({key:"value"}, (err, data) => {
  if (!err)
    console.log(data)
})

Warning: While stringifyAsync is in progress (i.e. before the callback is executed), it is the user's responsibility to ensure that the Javascript object value (or any of its child objects) is not modified in any way. Object modification between the async function invocation and issuing of the completion callback may lead to undefined behavior (and can possibly result in an inadvertent crash or object corruption).

parseAsync:

parseAsync(text, reviver, callback)

const yj = require('yieldable-json')
yj.parseAsync('{"key":"value"}', (err, data) => {
  if (!err)
    console.log(data)
})

GitHub Issues

This project uses GitHub Issues to track ongoing development and issues. Be sure to search for existing bugs before you create another one. Contributions are always welcome!

Collaborators

  • bidipyne - Bidisha Pyne <bidisha.pyne2015a@vit.ac.in> (she/her)
  • gireeshpunathil - Gireesh Punathil <gpunathi@in.ibm.com> (he/him)
  • HarshithaKP - Harshitha K P <harshi46@in.ibm.com> (she/her)
  • LakshmiSwethaG - Lakshmi Swetha Gopireddy <lakshmiswethagopireddy@gmail.com> (she/her)
  • mhdawson - Michael Dawson <michael_dawson@ca.ibm.com> (he/him)
  • sam-github - Sam Roberts <vieuxtech@gmail.com> (he/him)
  • sreepurnajasti - Sreepurna Jasti <sreepurna.jasti@gmail.com> (she/her)