@tokenizer/inflate

@tokenizer/inflate

@tokenizer/inflate is a package designed for handling and extracting data from ZIP files efficiently using a tokenizer-based approach. The library provides a customizable way to parse ZIP archives and extract compressed data while minimizing memory usage.

Features

• Efficient Decompression: Handles streams compressed with DEFLATE and related formats (e.g., gzip).
• Tokenizer Compatibility: Seamlessly integrates with strtok3. For example, use @tokenizer/s3 for efficient partial extraction of a Zip stored on AWS S3 cloud file storage.
• Streamlined Interface: Provides an intuitive API for working with compressed data in streaming and random-access scenarios.
• Chunked Data Access: Leverages the underlying media's capabilities to offer chunked or random access to data, unlike traditional streams.
• Plug-and-Play: Easily integrate with existing tokenizer-based workflows for parsing file metadata or binary structures.
• Interrupt the extraction process conditionally.

Installation

npm install @tokenizer/inflate

Usage

Example: Extracting Specific Files

The following example demonstrates how to use the library to extract .txt files and stop processing when encountering a .stop file.

import { ZipHandler } from '@tokenizer/inflate';
import { fromFile } from 'strtok3';

const fileFilter = (file) => {
  console.log(`Processing file: ${file.filename}`);

  if (file.filename?.endsWith(".stop")) {
    console.log(`Stopping processing due to file: ${file.filename}`);
    return { handler: false, stop: true }; // Stop the unzip process
  }

  if (file.filename?.endsWith(".txt")) {
    return {
      handler: async (data) => {
        console.log(`Extracted text file: ${file.filename}`);
        console.log(new TextDecoder().decode(data));
      },
    };
  }

  return { handler: false }; // Ignore other files
};

async function extractFiles(zipFilePath) {
  const tokenizer = await fromFile(zipFilePath);
  try {
    const zipHandler = new ZipHandler(tokenizer);
    await zipHandler.unzip(fileFilter);
  } finally {
    await tokenizer.close();
  }
}

extractFiles('example.zip').catch(console.error);

API

ZipHandler

A class for handling ZIP file parsing and extraction.

Constructor

new ZipHandler(tokenizer: ITokenizer, options?: ZipHandlerOptions)

• tokenizer: An instance of ITokenizer to read the ZIP archive.
• options: Optional processing limits and traversal settings. Each numeric limit accepts a positive safe integer or Infinity. Omitted limits default to Infinity, preserving unlimited processing.

const zipHandler = new ZipHandler(tokenizer, {
  maxEntryCount: 1024,
  maxTotalScanBytes: 16 * 1024 * 1024,
  maxTotalDataDescriptorScanBytes: 1024 * 1024,
  maxSkippedEntrySize: 1024 * 1024,
  maxDataDescriptorScanBytes: 1024 * 1024,
  maxCompressedSize: 1024 * 1024,
  maxUncompressedSize: 4 * 1024 * 1024
});

maxDataDescriptorScanBytes bounds bytes before the descriptor signature when the compressed size is unknown, even for ignored entries. Descriptor candidates are checked against their compressed-size field so an embedded signature with a different size does not end the scan. Lookahead uses up to 16 additional bytes to validate a candidate descriptor. maxCompressedSize bounds compressed data buffered for selected entries, and maxUncompressedSize bounds their decompressed output, including stored entries. Limits apply to local-header and central-directory sizes before allocation, and to actual output before retaining chunks or combining buffers. Decompression is canceled on overflow. maxEntryCount bounds entries visited, including ignored entries. maxTotalScanBytes bounds metadata and compressed data consumed across one traversal, including ignored local-header payloads. Random-access seek distances are excluded; the end-of-directory search and record are included, and bytes read again during extraction count again. Ignored central-directory payloads are not read or counted. maxTotalDataDescriptorScanBytes bounds cumulative payload bytes scanned for unknown-size entries, excluding their descriptors. maxSkippedEntrySize bounds a single ignored payload when local-header traversal must pass over it. Counters reset for each unzip() or readCentralDirectory() call.

These limits are not a total process memory budget; tokenizer lookahead and native decompression have their own buffers. Configure tokenizer limits separately where available.

A limit violation throws the exported ZipLimitError (a RangeError), with limit, requestedSize, and maxSize properties. Sizes are byte counts except for maxEntryCount. No data handler is called for an entry rejected by a limit; earlier handlers may already have run. Invalid numeric limit options throw RangeError when constructing the handler. Invalid non-boolean values for useCentralDirectory or allowTruncatedDataDescriptor throw TypeError during construction. After a processing failure, close the tokenizer; extraction is not resumable.

For random-access sources, central-directory entries are parsed within the directory's declared byte range using a borrowed tokenizer view. An entry crossing that range throws strtok3.TokenizerBoundaryError. Reading the directory restores the source position on both success and failure, and leaves the source open. unzip() processes directory entries one at a time and stops parsing when the callback returns stop: true, restoring the original source position on completion or failure. readCentralDirectory() still explicitly collects an array.

Set useCentralDirectory: false to scan local headers even on random-access sources. This is useful for detectors that need consistent local-header ordering and support partial archives. It does not disable entry or byte limits.

allowTruncatedDataDescriptor: true lets a detector receive the available entry data when a descriptor is missing or truncated at EOF. Extraction still rejects the incomplete descriptor after the handler returns; the caller can retain a detection result. All configured limits still apply. The default is false, which rejects incomplete descriptors before delivering data.

Methods

• isZip(): Promise<boolean>

  Determines whether the input file is a ZIP archive.

• unzip(fileCb: InflateFileFilter): Promise<void>

  Extracts files from the ZIP archive, applying the provided InflateFileFilter callback to each file. To extract a file, return an object whose handler is an InflatedDataHandler. Return { handler: false } to skip the file.

Types

InflateFileFilter

type InflateFileFilter = (file: ILocalFileHeader) => InflateFileFilterResult;

Callback function that receives the ZIP entry header and returns an InflateFileFilterResult to select or skip the entry.

InflateFileFilterResult

type InflateFileFilterResult = {
  handler: InflatedDataHandler | false; // Handle file data or ignore
  stop?: boolean; // Stop processing further files
};

Returned from InflateFileFilter to control file handling and extraction flow. Set handler to an InflatedDataHandler to receive the entry data, or false to skip it. Set stop: true to stop after the current entry has been handled or skipped.

InflatedDataHandler

type InflatedDataHandler = (fileData: Uint8Array) => Promise<void>;

Asynchronous callback assigned to InflateFileFilterResult.handler for a selected entry. It receives the complete uncompressed entry data as a Uint8Array, including for stored (uncompressed) entries. Extraction awaits the returned promise before processing the next entry.

Compatibility

This module is a pure ECMAScript Module (ESM). The distributed JavaScript codebase is compliant with the ECMAScript 2020 (11th Edition) standard. If used with Node.js, it requires version ≥ 18.

Contributing

See the contribution guide for code style and development checks.

License

This project is licensed under the MIT License. See the LICENSE file for details.