# hdf5-indexed-reader

> Module based on jsfive for efficient remote access of very large HDF5 files

Latest version **1.0.1** (published 2024-08-23) · MIT license · 0 weekly downloads

## Install

```sh
npm install hdf5-indexed-reader
pnpm add hdf5-indexed-reader
yarn add hdf5-indexed-reader
bun add hdf5-indexed-reader
```

## Health

**Score 20/100 (F)** — status: abandoned.

Positive: esm support; no vulnerabilities.

Warnings: low downloads; no types.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.1 |
| Published | 2024-08-23 |
| First published | 2024-08-23 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | ESM |
| Dependencies | 0 |
| Unpacked size | 445 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Jim Robinson |
| Maintainers | igvteam |
| Keywords | hdf5, javascript, es6, browser |

## Links

- npm: https://www.npmjs.com/package/hdf5-indexed-reader
- Repository: https://github.com/jrobinso/hdf5-indexed-reader
- Issues: https://github.com/jrobinso/hdf5-indexed-reader/issues
- npm.io page: https://npm.io/package/hdf5-indexed-reader

## Alternatives

- [@opentelemetry/exporter-zipkin](https://npm.io/package/@opentelemetry/exporter-zipkin.md) — 14.8M weekly downloads
- [pusher-js](https://npm.io/package/pusher-js.md) — 2.0M weekly downloads
- [browserify](https://npm.io/package/browserify.md) — 1.7M weekly downloads
- [sqs-consumer](https://npm.io/package/sqs-consumer.md) — 1.7M weekly downloads
- [@sanity/eventsource](https://npm.io/package/@sanity/eventsource.md) — 930.8K weekly downloads

## Recent versions

- 1.0.1 (latest) — 2024-08-23
- 1.0.0 — 2024-08-23

## README

# hdf5-indexed-reader

## Summary

hdf-indexed-reader is a module for efficient querying of HDF5 files over the web. It enables loading of individual
datasets from remote files without the need to load the entire file into memory. It works in
conjunction with the companion project [hdf5-indexer](https://github.com/jrobinso/hdf5-indexer), which annotates
HDF5 files with an index mapping object path names to file offsets.

The module is built on a fork of [jsfive](https://github.com/usnistgov/jsfive). The fork is available at
https://github.com/jrobinso/hdf5-indexed-reader.

## Motivation

The driving use case for this project involves extracting individual datasets for visualization in a web browser
from large HDF5 files (~200 GB) containing 10s of thousands of individual datasets. Loading such files over
the web with available solutions present 2 problems

* The file is too large to load into browser memory in its entirety

* Finding the file offset for the object desired involves walking a linked list of nodes of containing and sibling
  objects
  These nodes can be located anywhere in the file, resulting in an explosion of http range requests which can quickly
  freeze the application.

This project addresses these issues by (1) using range queries to load slices of the file as needed, and (2) supporting
a pre-built index for mapping object (groups and datasets) paths to file offsets, negating the need to walking the
linked list of container objects to build the index at runtime..

## Limitations

* As this project is based on [jsfive](https://github.com/usnistgov/jsfive), some limitations of that tool apply here,
  namely not all datatypes are supported.

* This reader is designed for large HDF5 files containing many datasets. Small files will likely not  
  benefit from indexing and incremental loading. Additionally, the benefit of indexing is reduced if the number
  of datasets is small.

## Build

```
npm run install
npm run build
```

The build creates 3 packages

* hdf5-indexed-reader.esm.js - an ES module for use in a web browser
* hdf5-indexed-reader.node.cjs - a common JS module for use with Node
* hdf5-indexed-reader.node.mjs - an ES module for use with Node

## Usage

The module exports a single function, ```openH5File( {options} )```. The HDF5 file is specified with one of the
following
properties

* url - url to the hdf5 file
* path - local file path, **Node only**
* file - browser `File` or other `Blob` like object

URL fetches are cached to avoid separate individual requests for small amounts of data. The following optional
properties controls
the cache

* fetchSize - minimum size in bytes for each http request. Defaults to 2000  (2 kb)
* maxSize - the maximum number of bytes to cache. Default value is 200000  (200 kb)

In cases where it is not possible to modify the HDF5 file [hdf5-indexer](https://github.com/jrobinso/hdf5-indexer) can
create an external index as a json file.  This file can be used with one of the following properties

* indexURL - url to index json file
* indexPath - local file path, `node` only
* indexFile - browser `File` object

## Example

Load a `Dataset` from a remote HDF5 file and fetch its shape, data type, and values.

```js
import {openH5File} from "dist/esm/hdf5-indexed-reader.esm.js"

const hdfFile = await openH5File({
    url: "https://www.dropbox.com/s/53fbs3le4a65noq/spleen_1chr1rep.indexed.cndb?dl=0",
})

const spatialPostionDataset = await hdfFile.get('/replica10_chr1/spatial_position/1149')
const shape = await spatialPostionDataset.shape
const dtype = await spatialPostionDataset.dtype
const values = await spatialPostionDataset.value

```

See examples folder for node cjs and es examples.

---
_Source: https://npm.io/package/hdf5-indexed-reader · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._