# flumelog-offset

> a flumelog based on offset into a file

Latest version **3.4.4** (published 2020-01-13) · MIT license · 0 weekly downloads

## Install

```sh
npm install flumelog-offset
pnpm add flumelog-offset
yarn add flumelog-offset
bun add flumelog-offset
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 3.4.4 |
| Published | 2020-01-13 |
| First published | 2016-11-07 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 10 |
| Unpacked size | 25.2 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 10 |
| Author | 'Dominic Tarr' |
| Maintainers | arj03, christianbundy, dominictarr, regular |

## Links

- npm: https://www.npmjs.com/package/flumelog-offset
- Repository: https://github.com/flumedb/flumelog-offset
- Issues: https://github.com/flumedb/flumelog-offset/issues
- npm.io page: https://npm.io/package/flumelog-offset

## Dependencies (10)

- [obv](https://npm.io/package/obv.md) 0.0.1
- [int53](https://npm.io/package/int53.md) ^1.0.0
- [looper](https://npm.io/package/looper.md) ^4.0.0
- [hashlru](https://npm.io/package/hashlru.md) ^2.3.0
- [uint48be](https://npm.io/package/uint48be.md) ^2.0.1
- [pull-cursor](https://npm.io/package/pull-cursor.md) ^3.0.0
- [pull-looper](https://npm.io/package/pull-looper.md) ^1.0.0
- [pull-stream](https://npm.io/package/pull-stream.md) ^3.6.13
- [append-batch](https://npm.io/package/append-batch.md) 0.0.2
- [aligned-block-file](https://npm.io/package/aligned-block-file.md) ^1.2.0

## Recent versions

- 3.4.4 (latest) — 2020-01-13
- 3.4.3 — 2019-07-16
- 3.4.2 — 2019-05-14
- 3.4.1 — 2019-05-07
- 3.4.0 — 2019-04-25
- 3.3.2 — 2018-09-22
- 3.3.1 — 2018-06-06
- 3.3.0 — 2018-05-31
- 3.2.7 — 2018-05-17
- 3.2.6 — 2017-10-20
- 3.2.5 — 2017-10-14
- 3.2.4 — 2017-10-14
- 3.2.3 — 2017-09-22
- 3.2.2 — 2017-08-03
- 3.2.1 — 2017-07-29
- … 15 more at https://npm.io/package/flumelog-offset/versions

## README

# flumelog-offset

An flumelog where the offset into the file is the key.
Each value is appended to the log with a double ended framing,
and the "sequence" is the position in the physical file where the value starts,
this means if you can do a read in O(1) time!

Also, this is built on top of [aligned-block-file](https://github.com/flumedb/aligned-block-file)
so that caching works very well.

## Usage

initialize with a file and a codec, and wrap with flumedb.

``` js
var OffsetLog = require('flumelog-offset')
var codec = require('flumecodec')
var Flume = require('flumedb')

var db = Flume(OffsetLog(filename, {codec: codec.json}))
  .use(...) //also add some flumeviews

db.append({greets: 'hello!'}, function (cb) {

})

```

## Options

```
var OffsetLog = require('flumelog-offset')
var log = OffsetLog('/data/log', {
  blockSize: 1024,        // default is 1024*16
  codec: {encode, decode} // defaults to no codec, expects buffers. for json use flumecodec/json
  flags: 'r',             // default is 'r+' (from aligned-block-file)
  cache: {set, get}       // default is require('hashlru')(1024)
  offsetCodec: {          // default is require('./frame/offset-codecs')[32]
    byteWidth,            // with the default offset-codec, the file can have
    encode,               // a size of 4GB max.
    decodeAsync
  }
})
```

## legacy

if you used `flumelog-offset` before 3, and want to read your old
data, use `require('flumelog-offset/legacy')`


## recovery

If your system crashes while an append is in progress, it's unlikely
but possible to have a partially written state. `flumelog-offset`
will rewind to the last good state on the next start up.

After running this for several months (in my personal secure-scuttlebutt
instance) I eventually got an error, which lead to the changes
in this version.

## format

data is stored in a append only log, where the byte index
of the start of a record is the primary key (`offset`).

```
offset-><data.length (UInt32BE)>
        <data ...>
        <data.length (UInt32BE)>
        <file_length (UInt32BE or Uint48BE or Uint53BE)>
```
by writing the length of the data both before and after each record
it becomes possible to scan forward and backward (like a doubly linked list)

It's very handly to be able to scan backwards, as often you want
to see the last N items, and so you don't need an index for this.

## future ideas

* secured file (hashes etc)
* encrypted file
* make the end of the record be the primary key.
  this might make other code nicer...

## License

MIT

---
_Source: https://npm.io/package/flumelog-offset · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._