@himpun/collator v0.0.1

Introduction

Collator is a javascript library used to organize complex data. It is intended to be used as a key encoder for key value store.

If you want to store your array key in mixed order while taking advantage of the sorted key nature of the key value store, then this library is for you.

Getting started

In order to start using this library, you first need to install it from NPM registry.

npm install @himpun/collator

You can then import it as follows.

const collator = require("@himpun/collator");

In order to encode key, you can do so;

const { asc, desc } = collator.order;

// The first element is sorted in ascending order and the second element is sorted in descending order
// Every element if not specified is by default sorted in ascending order
const timeKey = ["time", desc(Date.now())];
const encodedTimeKey = collator.encode(timeKey);

// You can of course do even more complex sorting where the first, second and third element is in descending, ascending and descending order respectively.
for (let i = 0; i < 10; i++) {
  for (let j = 0; j < 10; j++) {
    for (let k = 0; k < 10; k++) {
      const complexKey = [desc(i), asc(j), desc(k)];
      const encodedComplexKey = collator.encode(complexKey);
    }
  }
}

In order to decode key, you can do so;

const decodedTimeKey = collator.decode(encodedTimeKey);
const decodedComplexKey = collator.decode(encodedComplexKey);

Sort order

The library sort the elements in the following order;

asc  - null
asc  - false
asc  - true
asc  - NaN
asc  - number
asc  - buffer
asc  - string
asc  - array
asc  - object
asc  - undefined
desc - undefined
desc - object
desc - array
desc - string
desc - buffer
desc - number
desc - NaN
desc - true
desc - false
desc - null

Example

This library currently natively support as key encoder for both FoundationDB and LevelDB. If you need to use it as a key encoder for other key value store, maybe you can write a wrapper for that or open an issues for me.

In order to use this library with FoundationDB, you can do so;

const collator = require("@himpun/collator");
const foundationdb = require("foundationdb");
foundationdb.setAPIVersion(620);

const database = foundationdb
  .open()
  .withKeyEncoding(collator)
  .withValueEncoding(foundationdb.encoders.json);

const { asc, desc } = collator.order;
(async function main() {
  const data = await database.doTransaction(async (tx) => {
    tx.clearRange(asc(null), desc(null));
    for (let i = 0; i < 3; i++) {
      for (let j = 0; j < 3; j++) {
        for (let k = 0; k < 3; k++) {
          const complexKey = [desc(i), asc(j), desc(k)];
          const value = { message: "hello, world" };
          tx.set(complexKey, value);
        }
      }
    }

    const collection = [];
    for await (const [key, value] of tx.getRange(asc(null), desc(null))) {
      collection.push({ key, value });
    }
    return collection;
  });
  console.info(data);
})().catch(console.error);

In order to use this library with LevelDB, you can do so;

const collator = require("@himpun/collator");
const encode = require("encoding-down");
const leveldown = require("leveldown");
const levelup = require("levelup");

const encoding = { keyEncoding: collator, valueEncoding: "json" };
const database = levelup(encode(leveldown("data"), encoding));

const { asc, desc } = collator.order;
(async function main() {
  await database.clear({ gt: asc(null), lte: desc(null) });

  for (let i = 0; i < 3; i++) {
    for (let j = 0; j < 3; j++) {
      for (let k = 0; k < 3; k++) {
        const complexKey = [desc(i), asc(j), desc(k)];
        const value = { message: "hello, world" };
        await database.put(complexKey, value);
      }
    }
  }

  const data = [];
  database
    .createReadStream({ gt: asc(null), lte: desc(null) })
    .on("data", (pair) => data.push(pair))
    .on("end", () => console.info(data));
})().catch(console.error);

Both of these example will returns all data inside the store as follows;

[
  { "key": [2, 0, 2], "value": { "message": "hello, world" } },
  { "key": [2, 0, 1], "value": { "message": "hello, world" } },
  { "key": [2, 0, 0], "value": { "message": "hello, world" } },
  { "key": [2, 1, 2], "value": { "message": "hello, world" } },
  { "key": [2, 1, 1], "value": { "message": "hello, world" } },
  { "key": [2, 1, 0], "value": { "message": "hello, world" } },
  { "key": [2, 2, 2], "value": { "message": "hello, world" } },
  { "key": [2, 2, 1], "value": { "message": "hello, world" } },
  { "key": [2, 2, 0], "value": { "message": "hello, world" } },
  { "key": [1, 0, 2], "value": { "message": "hello, world" } },
  { "key": [1, 0, 1], "value": { "message": "hello, world" } },
  { "key": [1, 0, 0], "value": { "message": "hello, world" } },
  { "key": [1, 1, 2], "value": { "message": "hello, world" } },
  { "key": [1, 1, 1], "value": { "message": "hello, world" } },
  { "key": [1, 1, 0], "value": { "message": "hello, world" } },
  { "key": [1, 2, 2], "value": { "message": "hello, world" } },
  { "key": [1, 2, 1], "value": { "message": "hello, world" } },
  { "key": [1, 2, 0], "value": { "message": "hello, world" } },
  { "key": [0, 0, 2], "value": { "message": "hello, world" } },
  { "key": [0, 0, 1], "value": { "message": "hello, world" } },
  { "key": [0, 0, 0], "value": { "message": "hello, world" } },
  { "key": [0, 1, 2], "value": { "message": "hello, world" } },
  { "key": [0, 1, 1], "value": { "message": "hello, world" } },
  { "key": [0, 1, 0], "value": { "message": "hello, world" } },
  { "key": [0, 2, 2], "value": { "message": "hello, world" } },
  { "key": [0, 2, 1], "value": { "message": "hello, world" } },
  { "key": [0, 2, 0], "value": { "message": "hello, world" } }
]

Support

This library is by no means done, it still need to be battle tested. Hence, any feedback on your end is highly appreciated. Two particular area that is on the top of my list is; correctness and performance.

In order to communicate, feel free to open an issues or even better create a pull request.

If this library is useful to you or your organization, please like this repository and consider buying me a cup of coffee.

License

MIT License

Copyright (c) 2020 Endy bin Jasmi

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.