@gmod/bgzf-filehandle

@gmod/bgzf-filehandle

Reads block-gzipped (BGZF) files, such as those created by bgzip, using coordinates from the uncompressed file. @gmod/indexedfasta, @gmod/bam and @gmod/tabix all read their BGZF through it.

Decompression runs on libdeflate, compiled to WebAssembly and inlined in the bundle as base64, so there is no .wasm file to serve. On this repo's fixtures it inflates two to three times faster than pako, and about twice as fast as the browser's own DecompressionStream — tables and reasoning, including why not just use DecompressionStream, in docs/optimizations.md.

Terms

The docs here lean on three words, and two of them mean something narrower than they sound:

• BGZF block — bgzip files are so called 'multi-member gzip files', basically indepently compressed gzip blocks that are concatenated together. https://www.htslib.org/doc/bgzip.html they are at most 64KB compressed, its trailer recording its own uncompressed length. Blocks decode independently of each other, enabling both random access and the worker pool. A bare "block" always means this one, never a deflate block.
• Virtual offset — {blockPosition, dataPosition}: which BGZF block, and how far into that block's decompressed bytes.
• Chunk — a range between two virtual offsets, {minv, maxv}; a BAM or tabix index resolves a query to one or more chunks. A chunk covers a run of consecutive BGZF blocks and usually starts and ends partway through the first and last of them.

Install

npm install @gmod/bgzf-filehandle

BgzfFilehandle

Read a bgzip-compressed file with a .gzi index as though it never was:

import { BgzfFilehandle } from '@gmod/bgzf-filehandle'
import { LocalFile } from 'generic-filehandle2'

const f = new BgzfFilehandle({
  filehandle: new LocalFile('path/to/my_file.gz'),
  gziFilehandle: new LocalFile('path/to/my_file.gz.gzi'),
  blockConcurrency: 10, // concurrent range requests per read, default 10
})

// read(length, position) — matches generic-filehandle2
const data: Uint8Array = await f.read(300, 0)

Create the index with bgzip -i my_file, or bgzip -r my_file.gz for an already-compressed one. Everything a read touches arrives as one contiguous range and inflates in one call.

When the file is remote, pass @gmod/range-cache-filehandle as the filehandle. Each of those contiguous ranges is then cached in 256 KiB chunks, so a later read that overlaps one already fetched is served from memory instead of the network.

unzip

Decompress a BGZF or plain gzip buffer, falling back to plain gzip if the input is not BGZF:

import { unzip } from '@gmod/bgzf-filehandle'

const decompressed: Uint8Array = await unzip(compressedData)

unzipChunkSlice

Decompress a range of BGZF blocks and slice out a virtual offset range — what BAM and tabix readers do with a chunk from a BAI/TBI:

import { MAX_BGZF_BLOCK_SIZE, unzipChunkSlice } from '@gmod/bgzf-filehandle'

const minv = { blockPosition: 1234, dataPosition: 56 }
const maxv = { blockPosition: 9876, dataPosition: 78 }

// input starts at minv.blockPosition and must run through the END of the block
// at maxv.blockPosition. That block's compressed length is recorded nowhere, so
// over-read by one maximum-size block to cover it
const compressedData = await filehandle.read(
  maxv.blockPosition + MAX_BGZF_BLOCK_SIZE - minv.blockPosition,
  minv.blockPosition,
)

const { buffer, cpositions, dpositions } = await unzipChunkSlice(
  compressedData,
  { minv, maxv },
)

buffer holds the decompressed bytes between the two offsets. cpositions and dpositions are Float64Array block boundaries in compressed and decompressed coordinates, for generating stable feature IDs across chunk boundaries.

Worker pool

unzipChunkSlice takes an optional pool that spreads a chunk's blocks across Web Workers — close to linear to about four workers. Bytes cross as transferable objects rather than copies, so the handoff costs the same whatever the size and needs no cross-origin isolation:

import { getSharedWorkerPool, unzipChunkSlice } from '@gmod/bgzf-filehandle'

const pool = await getSharedWorkerPool() // undefined if Workers are unavailable
const result = await unzipChunkSlice(compressedData, chunk, pool)

A chunk under 0.4MB inflated stays on the calling thread, where a worker costs more than it returns, so a shared pool starts its workers only once a larger one arrives. No read waits on them: a pool inflates on the calling thread while its workers boot, and for good if one of them fails.

Safe to pass unconditionally — undefined keeps the sequential wasm path, and readers that take a pool of their own (@gmod/bam's bgzfWorkerPool) accept the same value. Worker counts, cross-thread sharing, lifecycle and driving a pool directly: docs/worker-pool.md.

Docs

• docs/optimizations.md — benchmark numbers, and what we measured, kept and rejected
• docs/worker-pool.md — pool lifecycle, sizing, sharing, and scanBgzfBlocks
• CONTRIBUTING.md — development and release steps

Academic use

This package was written with funding from the NHGRI as part of the JBrowse project. If you use it in an academic project that you publish, please cite the most recent JBrowse paper, which will be linked from jbrowse.org.

License

MIT Robert Buels, Colin Diesh