3.0.2 • Published 4 months ago
cardex v3.0.2
cardex
Indexes for CARs.
Implementations of CARv2 indexes in JavaScript. Status:
- IndexSorted ✅ (complete & compatible)
- MultihashIndexSorted ✅ (complete & compatible)
Install
npm install cardex
Usage
Write index
import fs from 'fs'
import { Readable } from 'stream'
import { CarIndexer } from '@ipld/car/indexer'
import { IndexSortedWriter } from 'cardex'
const carStream = fs.createReadStream('my.car')
const indexer = await CarIndexer.fromIterable(carStream)
const { readable, writable } = new TransformStream()
const writer = IndexSortedWriter.createWriter({ writer: writable.getWriter() })
readable.pipeTo(Readable.toWeb(fs.createWriteStream('my.car.idx')))
for await (const { cid, offset } of indexer) {
await writer.add(cid, offset)
}
await writer.close()
Read index
import fs from 'fs'
import { Readable } from 'stream'
import { IndexSortedReader } from 'cardex'
const carStream = fs.createReadStream('my.car.idx')
const reader = IndexSortedReader.createReader({ reader: Readable.toWeb(carStream).getReader() })
while (true) {
const { done, value } = await reader.read()
if (done) break
console.log(`${Buffer.from(value.digest).toString('hex')} @ ${value.offset}`)
}
Multi-index index
The multi-index index is a custom index allowing multiple CAR indexes to be grouped together in a single index.
Write multi-index
import { MultihashIndexSortedWriter } from 'cardex'
import { MultiIndexWriter } from 'cardex/multi-index'
const { readable, writable } = new TransformStream()
const writer = MultiIndexWriter.createWriter({ writer: writable.getWriter() })
readable.pipeTo(new WritableStream()) // destination
writer.add(carCID0, async ({ writer }) => {
const index0 = MultihashIndexSortedWriter.createWriter({ writer })
index0.add(cid, offset)
await index0.close()
})
writer.add(carCID1, async ({ writer }) => {
const index1 = MultihashIndexSortedWriter.createWriter({ writer })
index1.add(cid, offset)
await index1.close()
})
await writer.close()
Read multi-index
import { MultihashIndexSortedReader, IndexSortedReader } from 'cardex'
import { MultiIndexReader } from 'cardex/multi-index'
const readable = new ReadableStream() // reader of a multi-index bytes
const reader = MultiIndexReader.createReader({ reader: readable.getReader() })
// add readers to the multi-index reader (to allow reading different index types)
reader.add(MultihashIndexSortedReader)
reader.add(IndexSortedReader)
while (true) {
const { done, value } = await reader.read()
if (done) break
const { origin, multihash, digest, offset } = value // (origin is a CAR CID)
console.log(`${origin} -> ${CID.createV1(raw.code, multihash)} @ ${offset}`)
}
Universal reader
The universal reader is for when you don't know what type of CARv2 index you're reading. The universal reader automatically instantiates the correct reader for a given index based on the codec:
import fs from 'fs'
import { Readable } from 'stream'
import { UniversalReader } from 'cardex/universal'
const carStream = fs.createReadStream('my.car.idx')
const reader = UniversalReader.createReader({ reader: Readable.toWeb(carStream).getReader() })
while (true) {
const { done, value } = await reader.read()
if (done) break
console.log(`${Buffer.from(value.digest).toString('hex')} @ ${value.offset}`)
// Note: `value` might have `multihash` if reading from MultihashIndexSorted
// and it might have `origin` if reading from `MultiIndex`.
}
Contributing
Feel free to join in. All welcome. Open an issue!
License
Dual-licensed under MIT + Apache 2.0
3.0.2
4 months ago
3.0.1
4 months ago
3.0.0
4 months ago
2.3.3
5 months ago
2.3.2
5 months ago
2.3.1
10 months ago
2.3.0
10 months ago
2.2.1
11 months ago
2.2.3
11 months ago
2.2.2
11 months ago
2.2.4
11 months ago
2.2.0
11 months ago
2.1.0
11 months ago
2.0.1
11 months ago
2.0.0
11 months ago
1.0.1
1 year ago
1.0.0
2 years ago
0.0.0
2 years ago