Time-object-db NPM

title: Time Object Database

author: Claudio Heidel Schemberger

Welcome to time-object-db

Current version: 1.0.9

Challenge:

The challenge was to create a persistence system to store objects correlated with the date-time for later use without the need to install and maintain an exclusive service (external databases).

This persistence system need to have a quickly search feature, for this reason we use binary tree and cascading indexes.

At the same time this solution should be easy to deploy in cloud services such as the AWS Lambda functions (high availability and auto scaling).

The main features of the time-object-db are:

Create and delete databases
Create metrics collections and store objects.
Fetch metric objects from a period time (between fromEpoc and toEpoc).
Delete metric objects from a specific time value.
Complete clean a metric collection

Solution:

npm.io

time-object-db use single files in order to persist the information splitted in three levels:

Level 1: YYYY-MM (Max 12 folders per year)
Level 2: DD (Max 365 folders per parent)
Level 3: One file each 5 minutes - (Max 144 files per parent) (configurable)

Exist an index file per each Level 1 and 2 folders, these indexes help to find existing Level 3 files and prevent to use a full-scan to find the existing files.

How to Scale:

npm.io

Self Cluster Load Balancer:

If you enamble the cluster option, see the config.json file, the below load balancer feature is included out-of-the-box.

npm.io

Events:

This database emit 7 events

database/{databaseId}/database-delete => When a database is deleted
database/{databaseId}/metric-delete/{metridId} => When a metric is deleted
database/{databaseId}/index/{metridId} => When a index is updated or created
database/{databaseId}/delete/{metridId} => When an object is deleted
database/{databaseId}/insert/{metridId} => When a new object is inserted
database/{databaseId}/read/{metridId} => When a new query has been executed.

Development notes:

time-object-db is developed using only node.js

I'm developing this project in my free time from different places. For this reason, I use Repl.it, an online coding platform that allows me continue working regardless where I'm.

How to install

mkdir test-folder
cd test-folder
git clone https://github.com/mrcheidel/time-object-db.git
cd time-object-db
npm install
node index.js

How to use - From Node.js:

Run this demo

const toDb = require ("./todb.js");

const metricId = 'myMetricId';
//Declare some object to insert
const myObj1 =  {place: 'Parque del Retiro', address: 'Jerónimos', lat: 40.4151922, log: -3.683704};
const myObj2 =  {place: 'Jardín Botánico', address: 'Plaza de Murillo, 2', lat: 40.4133796, log: -3.688833};

//async/await
(async() => {
  //Obtain a new Db
  let config = await toDb({}).createDb();
  console.log (config);

  //Instance Db
  let db = toDb({databaseId: config.databaseId, token: config.key});
  
  //Check Db healt
  let health = await db.checkHealth();
  console.log (health);

  //Insert the same object with different time
  let value1 = await db.insertObject(metricId,123456789,myObj1).catch (err => console.log (err));
  let value2 = await db.insertObject(metricId, 123456790, myObj2).catch (err => console.log (err));
  console.log (value1);
  console.log (value2);

  let result = await db.insertBulkObjects(metricId, 
                    [[123450000, myObj1],
                     [123450001, myObj2],
                     [123450002, myObj1],
                     [123450003, myObj2]]).catch (err => console.log (err));
  console.log ("Bulk Insert Result:");
  console.log (result);

  //Seach objects in a windows time
  let search = await db.findObjects(metricId, 123450000, 123456900).catch (err => console.log (err));
  console.log (search);

  //Delete object by key (tm)
  await db.deleteObject(metricId, 123456789).catch (err => console.log (err));

  //Delete The metric
  await db.deleteMetric(metricId).catch (err => console.log (err));

  //Delete all Db
  await db.deleteDb().catch (err => console.log (err));
})()

How to use - API Contract:

This database may be used directly from your own node.js code or via RESTful API calls.

View & Execute the API contract

npm.io

Postman Collection

You could be use the Postman tool in order to test this API.

Find the Postman collection and examples that you could be import into the postman.

Todo

List of pending points

Implement the checkIndex like a health functionality with re-creation index option.
Backup & Restore API.
Event propagation between diferent server instances
Server Sent Events API in order to expose events through continuous query approach.
health check endpoint.
Client code for Node.js.
Self Load Balancer.