0.5.1 • Published 6 months ago

jbod v0.5.1

Weekly downloads
-
License
MIT
Repository
github
Last release
6 months ago

NPM version JSR version

English | 中文

API Document\ JBOD encoding format\ Benchmark

JavaScript Binary Object Data

JavaScript Binary Serialization and deserialization lib. Support for more JS data types。Can be used for transmission, and storage. \ JavaScript 二进制序列化与反序列库。支持更多的 JS 数据类型,序列化后大小占用很小。可用于传输、和存储。

Inspired by ProtoBuf, JBOD is more flexible than ProtoBuf. More applicable to dynamically typed languages like JavaScript.

Features

More JavaScript data types
typeNotes
boolean
null
undefined
numberSupport NaN、-Infinity、+Infinity
bigint
Uint8Array
string
RegExp
Array
Object
SymbolThe significance is not significant, only the description attribute will be retained after conversion
ErrorKeep only the cause, code, message, and name attributes
Map
Set
Smaller binary data size
Data typeByte size (JSON)Byte size (JBOD)
int(0~2147483647)1~ 101~5
int (-1~-2147483648)2~ 111~5
double1~228
boolean4(true)、5(false)1
null41
string (Set n as the UTF-8 encoding length of the string)n+2n+(1~5)

The data encoded by JBOD.encode() is about 70% the size of JSON The data size of structured encoding is 20% ~ 40% of JSON View Structured Encoding

View simple code size comparison example

Usage

Node

npm install jbod

import JBOD from "jbod";
const u8Arr = JBOD.encode(data);
const decodedData = JBOD.decode(u8Arr).data;

Deno

import JBOD from "jsr:@asn/jbod";
const u8Arr = JBOD.encode(data);
const decodedData = JBOD.decode(u8Arr).data;

Browser

import JBOD from "https://esm.sh/jbod";
const u8Arr = JBOD.encode(data);
const decodedData = JBOD.decode(u8Arr).data;

Structured encoding

There are some scenarios where the data structure is fairly fixed, in which case transmitting type information can be quite redundant. For example, the object type has very space-consuming key names, and it also has a significant impact on performance in the JavaScript environment. In some scenarios, the keys are fixed, in which case ideally, the encoding should not retain the key information, only encode the values, and the decoder should decode the values based on the predefined structure, then restore the object data. This feature is inspired by ProtoBuf.

Struct data types

Type symboldescriptionjs type
dyI3232-bit Integer (Dynamic length encode by zigzag + varints )number
dyI6464-bit Integer (Dynamic length, encode by zigzag + varints )bigint
i3232-bit Integernumber
i6464-bit Integerbigint
f6464-bit Floatnumber
boolBooleanboolean
stringstring
binaryUint8Array
anyAny type
anyArrayArray elements can be of any type
anyRecordObject pieces and values can be of any type
regExpRegExp
errorError
mapMap
setSet
symbolsymbol

Any type: The any type has an extra byte to hold type information than the fixed type

Example Struct definition

Suppose you need to define the following data structure:

interface Data {
  name: string;
  count?: number;
  custom: any;
  list: number[];
  items: { key1: any; key2: any }[];
}

Defining structure:

const struct = StructCodec.define({
  name: { id: 1, type: "string" },
  count: { id: 2, type: "dyI32", optional: true }, // Optional field
  custom: { id: 111, type: "any" }, // Any type, or you can omit type
  list: { id: 3, repeat: true, type: "dyI32" },

  // Array of objects
  items: {
    id: 4,
    repeat: true,
    type: {
      key1: { id: 1, type: "any" },
      key2: { id: 2, type: "any" },
    },
  },
});
const rawObject = { name: "test", count: 9, custom: [1] };
const u8Arr = struct.encode(rawObject);

const decodedData = struct.decode(u8Arr).data;
console.log(decodedData);

Note that the id is used to map with the key name, it must be a positive integer, and it cannot be repeated。\ For the any type, you don't have to write the type. In this case, you could have also defined it like this:

const struct = StructCodec.define({
  name: { id: 1, type: "string" },
  count: { id: 2, type: "dyI32", optional: true },
  custom: 111,
  list: { id: 3, repeat: true, type: "dyI32" },
  items: {
    id: 4,
    repeat: true,
    type: { key1: 1, key2: 2 },
  },
});

The any type contains an extra byte to hold the type information, depending on your use case

Examples

Simple code size comparison example

import JBOD, { StructCodec } from "jbod";
import { Buffer } from "node:buffer";
function encodeJSON(data: any) {
  return Buffer.from(JSON.stringify(data));
}
export const objData = {
  disabled: false,
  count: 100837,
  name: "Documentation",
  dataStamp: 4 / 7,
  id: 876,
};

const anyStruct = StructCodec.define({ disabled: 1, count: 2, name: 3, dataStamp: 4, id: 5 });
const fixedStruct = StructCodec.define({
  disabled: { id: 1, type: "bool" },
  count: { id: 2, type: "dyI32" },
  name: { id: 3, type: "string" },
  dataStamp: { id: 4, type: "f64" },
  id: { id: 5, type: "dyI32" },
});

console.log(encodeJSON(objData).byteLength); // 96
console.log(JBOD.encode(objData).byteLength); // 67   (70% of JSON)
console.log(anyStruct.encode(objData).byteLength); // 38 (55% of JSON)
console.log(fixedStruct.encode(objData).byteLength); // 34 (35% of JSON)
0.5.1

6 months ago

0.4.9

1 year ago

0.4.8

1 year ago

0.4.7

1 year ago

0.4.6

1 year ago

0.5.0

1 year ago

0.4.5

1 year ago

0.4.4

1 year ago

0.4.3

1 year ago

0.4.2

1 year ago

0.4.1

1 year ago

0.4.0

1 year ago

0.3.0

1 year ago

0.2.1

1 year ago

0.2.0

1 year ago

0.1.0

1 year ago

0.0.3

1 year ago

0.0.2

2 years ago

0.0.1

2 years ago