# node-bitcask
> An append only, on-disk log database for storing and retrieving logs fast using in memory hashmaps.
Latest version **1.0.0-beta.4** (published 2022-09-07) · ISC license · 0 weekly downloads
## Install
```sh
npm install node-bitcask
pnpm add node-bitcask
yarn add node-bitcask
bun add node-bitcask
```
## Health
**Score 15/100 (F)** — status: abandoned.
Positive: no vulnerabilities.
Warnings: low downloads; no types; no esm support.
Negative: abandoned; low maintenance score.
## Facts
| | |
|---|---|
| Version | 1.0.0-beta.4 |
| Published | 2022-09-07 |
| First published | 2022-08-08 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 0 |
| Unpacked size | 35.7 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Author | Ankush Patil |
| Maintainers | proankush |
| Keywords | bitcask, logs, beta |
## Links
- npm: https://www.npmjs.com/package/node-bitcask
- npm.io page: https://npm.io/package/node-bitcask
## Alternatives
- [cli-color](https://npm.io/package/cli-color.md) — 3.4M weekly downloads
- [log](https://npm.io/package/log.md) — 1.3M weekly downloads
- [logstash-client](https://npm.io/package/logstash-client.md) — 4.5K weekly downloads
- [@nocobase/plugin-logger](https://npm.io/package/@nocobase/plugin-logger.md) — 2.0K weekly downloads
- [child-process-debug](https://npm.io/package/child-process-debug.md) — 695 weekly downloads
## Recent versions
- 1.0.0-beta.4 (latest) — 2022-09-07
- 1.0.0-beta.1 (beta) — 2022-08-08
- 1.0.0-beta.3 — 2022-08-19
- 1.0.0-beta.2 — 2022-08-15
## README
# Node-Bitcask  
## [Whats bitcask, you ask?](https://docs.riak.com/riak/kv/2.2.3/setup/planning/backend/bitcask/index.html)
>"Bitcask[^1] is an Erlang application that provides an API for storing and retrieving key/value data using log-structured hash tables that provide very fast access. The design of Bitcask was inspired, in part, by log-structured filesystems and log file merging." - riak docs
[**node-bitcask**](https://www.npmjs.com/package/node-bitcask) is a open source NodeJS implementation of the proposed storage engine. It is a log structured hash table.
Log structured: Data structure that allows append operations only, to utilise the sequential speeds of traditional mechanical hard drives.
Hash tables: In memory Key-Value pairs with O(1) read and write complexity.
node-bitcask allows asynchronous and synchronous operations on such a log structured hash table, where values are md5 verified, to store and provide data accurately. It also features resilience to power outage with snapshots, vertical scalability, space optimization by periodic compaction.
## Installation
---
Install via npm:
```properties
npm install node-bitcask
```
## Instantiation
---
Using node-bitcask is extremely simple. node-bitcask can be imported with ES5 `require()`.
```js
const nb = require('node-bitcask');
nb.put("zebra", "an African wild animal that looks like a horse, with black or brown and white lines on its body");
nb.get("zebra");
```
## API
---
- [Inserting data](#inserting-data)
- [Accessing data](#accessing-data)
- [Exporting the database](#Exporting-the-database)
- [Importing previous database](#Importing-previous-database)
- [Deleting a log](#Deleting-a-log)
- [Deleting the database](#Deleting-the-database)
- [Check if key exists](#Checking-if-key-exists)
- [Check if empty](#Checking-if-empty)
- [Getting total keys count](#Getting-total-keys-count)
- [Iterating over the keys](#Iterating-over-the-keys)
### **Inserting data**
Data can be simply stored with:
```js
put(key, data, callback)
```
`key` is unique entity String that can be used to refer back to the `data`. Put returns void, and asynchronously stores the data.
**Note:** `putSync(key, data)` is also available for synchronous *put* operation.
### **Accessing data**
To get back your data use:
```js
get(key, callback)
```
`get` asynchronously find `key` referenced data and on success, provides the data to given callback. In case no data is found (maybe due to deleted key, incomplete storage, power-outage, bugs etc) `callback` will be invoked with `null` argument.
**Note:** `getSync(key)` is also available for synchronous *get* operation.
### **Exporting the database**
To export the database essential files
```js
exportDataSync(newLogFileDir, newKVFileDir)
```
`logFile` is an essential file which contains the entire data of the database.
`kvFile` keeps a snapshot of the in memory kv-store, just in case a power-outage occurs, or anything else goes wrong.
exportsDataSync accepts two `fs.PathLike` arguments which are used to copy the data to.
It synchronously copies all the data to given paths.
### **Importing previous database**
To import previously exported data use:
```js
importDataSync(oldLogFileDir, oldKVFileDir)
```
`oldLogFileDir` and `oldKVFileDir` are paths to where kvfile and logfiles were copied to. `importDataSync` first synchronously copies the data to its desired directory. After the copying succeeds the entire database is reconstructed from these files.
### **Deleting a log**
```js
deleteLog(key)
```
Deletes the key.
**Note:** after deletion, the data may still exist in the logfile for a small duration.
### **Deleting the database**
To delete all the data and the KV store use:
```js
nb.unload()
```
### **Checking if key exists**
To check if nb contains a undeleted key, use:
```js
nb.contains(key)
```
returns `true` if `key` is present in nb.
### **Checking if empty**
```js
nb.isEmpty()
```
returns `true` if nb is empty. This can be due no log operation executed, or every key is deleted.
### **Getting total keys count**
```js
nb.size()
```
Returns an Integer which equals to the count of active keys in nb.
### **Iterating over the keys**
nb allows reading sequentially over all the active keys.
```js
const keys = nb.keys()
for(let key of keys){
console.log(nb.getSync(key));
}
```
will iterate over all keys in nb and read them synchronously
## Configuration
---
node-bitcask allows for greater configuration of where data is stored, when to do compaction, etc. Configuring node-bitcask is as simple as:
```js
const nb = require("node-bitcask");
nb.configure({
dataDir: "./some/arbitrary/folder",
kvSnapshotPath: "./path/to/file",
backupKVInterval: 1000, //in ms
compactionInterval: 10000, //in ms
})
```
You can omit any key that you dont want to configure, and its value will stay to default.
## Some special notes
---
- Old time fans will remember there used to be Stream api for reading and writing large data with putStream and getStream. Sadly these features had to go, as hashing this large stream of data wouldn't be possible.
- *async fs* operations are handled by thread pool in nodejs. So performance of these operations will depend on the count of CPU cores in the system, and their respective speed. i.e. A single logical cpu core can do one fs operation at a time, so 4c/8t cpu will handle 8 fs operations concurrently.[^4]
- Why not cache frequently accessed keys? Good observation, still cache system like redis will be helpful only if it is run on another physical machine. Running it concurrently will affect the file system access performance of node-bitcask, since -as mentioned already- nodejs file system operations utilize thread pool, one less cpu thread will marginally drop the performance of node-bitcask. Integrating node-bitcask on a completely different stand alone hardware might be better choice, always.
## Known Issues
---
- ~~Sync read and writes will fail during compaction. Please prefer using async variations of both until future updates.~~
- Some writes between creating snapshot and powerloss/SIGTERM will be available in logfile, but no references will be written in snapshot.
- Current storage format for data can be improved.
## Changelogs
---
**Version 1.0.0-beta.4**
- Optimised write performance when consecutively writing to the same key asynchronously. Also decreased fs access for the same. Prefer using async operations.
- Faster reads when reading immediately after async write.
- Fix sync operations during compaction.
- Closing file descriptor opened during `getSync()`
**Version 1.0.0-beta.3**
- Compaction fix for data exceeding writeStream highwatermark(1kb).
- Queueing async get() and log() until compaction ends for consistency.
- Fix deletion for already deleted keys.
- Added helpfull utils like `isEmpty()`, `keys()`, `contains(key)`, `size()` for better manipulation and querying.
- Persisting garbage collectible information, so compaction can be efficient on imported database.
- Known issues section in README.
**Version 1.0.0-beta.2**
- Removed stream api.
- Added md5 checksums (crypto module) for verifying the integrity of written data.
- Replace setTimeout with setInterval for compaction.
- Fixed `WriteStream` used internally for compaction would not emit *close* event, also replace listener from *close* to *drain* event.
- Convert basic operations to asynchronous operations.
- Added synchronous functions.
- Added changelogs to README
- Some more minor fixes.
**Version 1.0.0-beta.1**
- Fixed undefined instance variables of node-bitcask resulted in failure of most operations
- Removed test operations from index.js which would get executed on importing node-bitcask.
- Handled file doesn't exist.
- Made internal variables and function private.
- Removed most of the debug logs
## References
---
1. Bitcask - Riak Docs. [^1]
2. Designing Data Intensive Applications by Martin Kleppmann.(came across bitcask reading this one, great book) [^2]
3. Bitcask Paper. [^3]
[^1]: https://docs.riak.com/riak/kv/2.2.3/setup/planning/backend/bitcask/index.html
[^2]: https://www.amazon.in/Designing-Data-Intensive-Applications-Reliable-Maintainable-ebook/dp/B06XPJML5D
[^3]: https://riak.com/assets/bitcask-intro.pdf
[^4]: https://www.youtube.com/watch?v=zphcsoSJMvM
---
_Source: https://npm.io/package/node-bitcask · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._