@ja-ja/json-stream v0.2.0
@ja-ja/json-stream
Lightweight JSON stream parsing library, < 4KB ungzipped
Process huge json documents in small chunks to extract data without blowing up your memory
Installation
NPM
npm install @ja-ja/json-stream
Yarn
yarn add @ja-ja/json-stream
Unit tests
Run tests with yarn test
. I tried to have as many tests as possible, but no one's perfect. A lot of tests were extracted from JSONTestSuite (See also Parsing JSON is a minefield)
Documentation / Samples
API documentation can be found in the docs
folder, samples are located in the samples
folder
Roadmap
- BOM handling and UTF-16
Usage
Parsing a JSON string
The snippet below shows the equivalent of using JSON.parse
(of course in this case using JSON.parse
is actually much more efficient):
import { parser } from '@ja-ja/json-stream'
function jsonParse(jsonString) {
try {
const { write, end } = parser({
// Empty options, just parse the document without doing anything special
})
// The write() function takes a Uint8Array containing the raw JSON data
// It takes care of UTF-8 decoding
const buffer = new TextEncoder().encode(jsonString)
write(buffer)
// Calling end() returns the parsed document
return end()
} catch (error) {
// Handle parsing error
console.error(error)
throw error
}
}
Streaming a fetch
response
The snippet below shows how to parse a fetch()
response in a streaming manner:
import { parser } from '@ja-ja/json-stream'
async function streamFetchResponse(response, options) {
try {
const { write, end } = parser(options)
// Get a stream reader from the response body
const reader = response.body.getReader()
// Read some data when it's available
let buffer = await reader.read()
while (!buffer.done && buffer.value !== undefined) {
// Write each chunk of the response
write(buffer.value)
buffer = await reader.read()
}
// Calling end() returns the parsed document
return end()
} catch (error) {
// Handle parsing error
console.error(error)
throw error
}
}
Streaming a JSON file in Node.js
The snippet below shows how to parse a JSON file in a streaming manner in Node.js:
import { parser } from '@ja-ja/json-stream'
import { open } from 'node:fs/promises'
async function streamJsonFile(path, options) {
try {
const { write, end } = parser(options)
// Open the file
const buffer = new Uint8Array(1024)
const file = await open(path)
// Read some data when it's available
let res = await file.read({ buffer })
while (res.bytesRead > 0) {
// Write each chunk of the response
write(buffer.subarray(0, res.bytesRead))
res = await file.read({ buffer })
}
file.close()
// Calling end() returns the parsed document
return end()
} catch (error) {
// Handle parsing error
console.error(error)
throw error
}
}
Ignoring elements
Ignoring an element means that while the JSON is still parsed and validated, no data will be created for that element, your options won't be executed for its children elements and nothing will be registered in the parent element:
import { parser } from '@ja-ja/json-stream'
const { write, end } = parser({
ignore: (path) => path.isKey(0, 'a'),
})
write(new TextEncoder().encode('{"a":1,"b":2}'))
const result = end()
// result === { b: 2 }
Getting raw JSON data
Much like when ignoring, the JSON is still parsed and validated, no data will be created for that element, your options won't be executed for its children elements, however the raw JSON data is recorded as the value (without whitespaces):
import { parser } from '@ja-ja/json-stream'
const { write, end } = parser({
raw: (path) => path.isKey(0, 'a'),
})
write(new TextEncoder().encode('{"a":1,"b":2}'))
const result = end()
// result === { a: Uint8Array([49]), b: 2 }
Transforming elements
You can act on created elements and transform them, replace them, etc. using the transform
function:
import { parser } from '@ja-ja/json-stream'
const { write, end } = parser({
transform: (path, value) => (path.isKey(0, 'a') ? 'tranformed' : value),
})
write(new TextEncoder().encode('{"a":1,"b":2}'))
const result = end()
// result === { a: 'transformed', b: 2 }
Dropping elements
Instead of ignoring entirely the element you can also parse it, and decide to drop it (which means the element won't be registered in its parent element and can be reclaimed by the garbage collector)
import { parser } from '@ja-ja/json-stream'
const { write, end } = parser({
drop: (path, value) => value === 1,
})
write(new TextEncoder().encode('{"a":1,"b":2}'))
const result = end()
// result === { b: 2 }
Parse an array, dropping elements as you go
import { parser } from '@ja-ja/json-stream'
const { write, end } = parser({
drop: (path, value) => {
if (path.length() === 1) {
// Handle the value
doStuff(value)
// Drop the value (don't push it in the root array)
return true
}
return false
},
})
write(new TextEncoder().encode('[1, 2, 3, 4, 5, "... my infinite array"]'))
const result = end()
// result === []