# @targetprocess/snapshot-serializer

> Library for big snapshot serialization/deserialization

Latest version **1.0.2** (published 2023-09-01) · ISC license · 0 weekly downloads

## Install

```sh
npm install @targetprocess/snapshot-serializer
pnpm add @targetprocess/snapshot-serializer
yarn add @targetprocess/snapshot-serializer
bun add @targetprocess/snapshot-serializer
```

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: has types; no vulnerabilities; high quality score.

Warnings: low downloads; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.2 |
| Published | 2023-09-01 |
| First published | 2023-02-15 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | bundled |
| Module format | CommonJS |
| Node | >=16.0.0 |
| Dependencies | 3 |
| Unpacked size | 25.9 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | rkozachenko@apptio.com |
| Maintainers | targetprocess-user |
| Keywords | zip, stream, json |

## Links

- npm: https://www.npmjs.com/package/@targetprocess/snapshot-serializer
- Repository: git@awsgitlab.tpondemand.net:sdk/snapshot-serializer
- npm.io page: https://npm.io/package/@targetprocess/snapshot-serializer

## Dependencies (3)

- [tmp](https://npm.io/package/tmp.md) ^0.2.1
- [loggerism](https://npm.io/package/loggerism.md) ^2.4.0
- [JSONStream](https://npm.io/package/JSONStream.md) ^1.3.5

## Alternatives

- [@mapbox/jsonlint-lines-primitives](https://npm.io/package/@mapbox/jsonlint-lines-primitives.md) — 5.3M weekly downloads
- [reftools](https://npm.io/package/reftools.md) — 3.5M weekly downloads
- [@hey-api/openapi-ts](https://npm.io/package/@hey-api/openapi-ts.md) — 3.5M weekly downloads
- [@mapbox/geojson-rewind](https://npm.io/package/@mapbox/geojson-rewind.md) — 2.4M weekly downloads
- [turbo-stream](https://npm.io/package/turbo-stream.md) — 1.7M weekly downloads

## Recent versions

- 1.0.2 (latest) — 2023-09-01
- 1.0.0 — 2023-02-15

## README

# snapshot-serializer

Big snapshot serialization/deserialization library

# Problem
Imagine that you need to serialize some object (aka snapshot), which has the following form:
```typescript
type Snapshot {
    metadata: {
        id: string
        author: string
        version: number
        // ... some other metadat properties
    },
    items: {
            id: number
            name: string,
            // ... some other item properties
        }[]
}
```

In most cases you can just do `JSON.stringify(snapshot)`.
But if you have a LOT of items you may face the following issues:
1. Total size of the result string length from `JSON.stringify(snapshot)` may exceed 512 MB, in that case you will face `"Invalid string length"` exception, caused by [String length limit in JS](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length)
2. Memory usage - all the snapshot object should be loaded to the memory.
3. The total size of stringified snapshot is very big.

# Solution
To solve the problem we split our initial type into the `Parts`. Each part can be either `Array` or `Object` type.

`Array` parts (in the example `items`) will be serialized per item and each serialized item written to file. Data for serialization provided using `AsyncGenerator` method, which allow to load data "by pages". Deserialization is performed by chunks - after `arrayBatchSize` items read, processing is started. After that, next chunk will be read and so on.
That allow to avoid `"Invalid string length"` exception and decrease memory consumption. 

`Object` parts (in the provided example that will be `metadata`) expected to have a limited size and will be serialized using regular `JSON.stringify` method to the file. Deserialization is performed using `JSON.parse` method.

# Dependencies
You need to install `zip` and `unzip` utilities on your system.

# Usage example

Please check [tests](./src/index.spec.ts) for usage example

---
_Source: https://npm.io/package/@targetprocess/snapshot-serializer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._