@muze-nl/od-jsontag

od-jsontag: On Demand JSONTag

od-jsontag is a lazy parser and serializer for large JSONTag-style object graphs. It stores data as one object per line, keeps references between objects as line numbers, and creates JavaScript Proxy objects so values are parsed only when your code actually touches them.

The goal is to let you work with large, connected data structures through a normal object API without paying the full parse cost up front.

import Parser from '@muze-nl/od-jsontag'
import serialize, {stringify} from '@muze-nl/od-jsontag/src/serialize.mjs'

const data = {
  articles: [
    {title: 'On demand parsing'}
  ]
}

const buffer = serialize(data)
const parser = new Parser()
const root = parser.parse(buffer)

// The root proxy exists immediately. Referenced objects are parsed on access.
console.log(root.articles[0].title)

console.log(stringify(buffer))

Why od-jsontag exists

od-jsontag was made for data that is too connected and too large to comfortably parse as one regular JSON document, but still wants to be used like ordinary JavaScript objects.

JSON is excellent when you want to load a complete tree. It is less ideal when:

• the file is large;
• most requests only touch a small part of the data;
• many objects reference the same object;
• the data needs to be shared between workers;
• identity matters, so repeated references should point at the same object;
• you want to preserve JSONTag metadata such as types and attributes.

od-jsontag approaches this by splitting a graph into lines. Line 0 is the root. Other lines are objects that can be referenced by ~1, ~2, and so on. The parser can scan the file and create lightweight proxies for each line, or it can use a supplied line index and skip the scan entirely. With an index, only the root line is parsed at first; referenced lines become proxies and parse lazily when accessed.

This is especially useful with SharedArrayBuffer, because the same serialized data can be shared with workers without copying the full object graph into each worker.

When to use it

Use od-jsontag when you have large, mostly-read object graphs and only a fraction of the objects are needed for a given operation.

Good fits:

• large JSONTag datasets where references and identity matter;
• read-heavy applications that open a large data file and inspect small parts;
• worker-based Node.js applications sharing data through SharedArrayBuffer;
• graph-like data where the same entity appears from many paths;
• data with JSONTag object attributes, typed values, or non-enumerable properties;
• applications that need access control hooks around object properties.

When not to use it

Do not use od-jsontag just because a file is JSON-shaped. For small files, JSON.parse or JSONTag.parse will usually be simpler and faster.

It is probably not the right fit when:

• you always need the full dataset immediately;
• your data is a plain tree with no shared references or identity concerns;
• you need broad query/filter/aggregate operations over millions of rows;
• you need a stable cross-language binary format;
• you need transactional updates, indexing, and persistence like a database.

For analytical workloads that scan a few fields across many similar objects, a columnar format or database may be a better match. od-jsontag is intentionally object-oriented: it optimizes lazy object access, not column scans.

Core ideas

One object per line

Serialized output is a newline-separated sequence of length-prefixed JSONTag values:

(23){"foo":[~1],"bar":[~2]}
(57)<object class="foo" id="1">{"name":"Foo",#"hidden":"bar"}
(57)<object class="bar" id="2">{"name":"Bar","children":[~1]}

Each line starts with (N), where N is the byte length of the JSONTag value after the prefix. References use ~lineNumber.

See docs/data-format.md for a simple explanation of the format.

Lazy proxies

parse() returns a proxy for the root value. Referenced objects are represented by proxies as well. The object body is parsed only when code reads a property, enumerates keys, checks in, defines a property, deletes a property, or performs another operation that needs the object contents.

const root = parser.parse(buffer)

// Parses root, then the referenced object at line 1.
console.log(root.foo[0].name)

Optional line index

If you already have an index of line number to byte positions, pass it as the second argument to parse():

const index = [
  [0, 28],
  [29, 98],
  [99, 161]
]

const root = parser.parse(buffer, JSON.stringify(index))

The index is an array or an object keyed by record number. Each entry is a [start, end] byte range in the data file, with an exclusive end. Sparse changeset indexes keep their original record numbers. It can be supplied as:

• an already parsed array or record-number object;
• a JSON string;
• a Uint8Array containing JSON.

The default parser has no Node imports. For indexed reads it also accepts a byte source with byteLength and a synchronous read(start, end) method that returns exactly that range as a Uint8Array. See portability for the source contract and runtime requirements.

For Node file descriptors and index-file paths, use the Node entry point:

import Parser from '@muze-nl/od-jsontag/src/node.mjs'
import {openSync, closeSync} from 'node:fs'

const parser = new Parser()
const dataFd = openSync('data.odjt', 'r')

try {
  const root = parser.parse(dataFd, 'data.odjt.index.json')
  console.log(root.foo[0].name)
}
finally {
  closeSync(dataFd)
}

Keep every source descriptor open, and its bytes unchanged, while the parser or its proxies are in use. The parser does not own or close descriptors. Repeated parse(fd, index) calls overlay records at their existing numbers; untouched records keep their previous file source. Buffer patches also work on indexed views. Use separate parsers for independently retained historical snapshots.

Read-only parsers cache up to 256 clean decoded records by default:

parser.cacheSize = 128
console.log(parser.cacheInfo())
parser.clearCache()

Evicted records reload when accessed; live object proxies keep their identity. The bound covers decoded record bodies, not the record index, caller-retained arrays/values, or edits. Mutable sessions retain touched records until the session is discarded or their updates are applied. See the reference for lifetime and error semantics.

Mutability

Parsers are immutable by default:

const parser = new Parser()
const root = parser.parse(buffer)

root.name = 'New name' // throws

Create a mutable parser by passing false as the second constructor argument or by setting parser.immutable = false:

const parser = new Parser(undefined, false)
const root = parser.parse(buffer)

root.name = 'New name'

Changed objects are serialized again. Unchanged parsed proxies can copy their original byte range back into the output.

Access control

You can install an access hook on parser.meta.access:

const parser = new Parser()

parser.meta.access = (object, property, method) => {
  return property === 'name'
}

const root = parser.parse(buffer)

console.log(root.name)  // allowed
console.log(root.secret) // undefined

The method argument is usually one of:

• get
• set
• has
• deleteProperty
• defineProperty

Access denial returns undefined or false, depending on the proxy operation.

Serialization

Use serialize(value, options) to create the od-jsontag byte representation. It returns a Uint8Array backed by a SharedArrayBuffer when available, or an ArrayBuffer otherwise. For large output, serializeChunks(value, options) yields framed Uint8Array chunks incrementally without allocating the entire serialized dataset. Both visit the complete record catalog, including records that have never been accessed.

const buffer = serialize(root)
const text = stringify(buffer)

Useful options:

• meta: share parser metadata such as resultArray and the id index.
• changes: true: serialize only changed lines as a patch-style stream.
• skipLength: true: internal option used when serializing a single line body.

See docs/reference.md for the API reference.

For benchmark results and memory tradeoffs compared with standard JSON, see docs/performance.md.

JSONTag compatibility

od-jsontag builds on @muze-nl/jsontag. It preserves JSONTag types and attributes for serialized values. It also adds support for non-enumerable object properties by prefixing the property with # inside the line format:

(57)<object class="foo" id="1">{"name":"Foo",#"hidden":"bar"}

Documentation

• API reference
• Data format
• Performance tradeoffs

Development

npm test
npm run lint
node --expose-gc benchmark/retention.mjs

Tests report coverage and fail on assertion failures. Incomplete coverage is explicitly allowed; coverage percentage is evidence, not a claim of completeness. The file-backed tests include unread records, sparse overlays, partial reads, Unicode, cache eviction, reflection policy and command-worker session resets.

See the hardening evaluation for verification against SimplyStore and the current operating limits.