njodb v0.4.38

njodb

njodb is a persistent, partitioned, concurrency-controlled, Node JSON object database. Data is written to the file system and distributed across multiple files that are protected by read and write locks. By default, all methods are asynchronous and use read/write streams to improve performance and reduce memory requirements (this should be particularly useful for large databases).

What makes njodb different than other Node JSON object databases?

Persistence - Data is saved to the file system so that it remains after the application that created it is no longer running, unlike the many existing in-memory solutions. This persistence also allows data to be made available to other applications.

Asynchronous and streaming - By default, all methods are asynchronous and non-blocking, and also use read and write streams, to make data access and manipulation efficient. Synchronous methods are also provided for those cases where they are desired or appropriate.

JSON records, not JSON files - Records are stored as individual lines of JSON objects in a file, so a read stream can be used to retrieve data rapidly, parse it in small chunks, and dispense with it when done. This removes the time and memory overhead required by solutions that store data as a single, monolithic JSON object. They must read all of that data into memory, and then parse all of it, before allowing you to use any of it.

Completely schemaless - While the JSON data itself is schemaless, it is also the case that data is not siloed into tables, or forced into collections, so the entire database, from top to bottom, is schemaless, not just the data. For a user or application, this means that there is no need to know anything about the database structure, only what data is being sought.

Balanced partitions - When inserting data, records are randomly distributed across partitions so that partition sizes are kept roughly equal, making data access times consistent. Manually resizing the database performs a similar distribution, so, as the database grows or shrinks, the partitions remain well-balanced.

Concurrency-controlled - File locks are used during read and write operations, ensuring data integrity can be maintained in a multi-user/multi-application environment. There are few, if any, existing solutions that are designed for such scenarios and that include this sort of data protection.

njodb even has its own command-line interface: check out njodb-cli.

Table of contents

• Install
• Test
• Introduction
• Constructor
  • Database properties
• Database management methods
  • stats / statsSync
  • grow / growSync
  • shrink / shrinkSync
  • resize / resizeSync
  • drop / dropSync
  • getProperties / setProperties
• Data manipulation methods
  • insert / insertSync
  • insertFile / insertFileSync
  • select / selectSync
  • update / updateSync
  • delete / deleteSync
  • aggregate / aggregateSync
• Finding and fixing problematic data

Install

npm install njodb

Test

npm test

Introduction

Load the module:

const njodb = require("njodb");

Create an instance of an NJODB:

const db = new njodb.Database();

Create some JSON data objects:

const data = [
    {
        id: 1,
        name: "James",
        nickname: "Good Times",
        modified: Date.now()
    },
    {
        id: 2,
        name: "Steve",
        nickname: "Esteban",
        modified: Date.now()
    }
];

Insert them into the database:

db.insert(data).then(results => /* do something */ );

Select some records from the database by supplying a function to find matches:

db.select(
    record => record.id === 1 || record.name === "Steve"
).then(results => /* do something */ );

Update some records in the database by supplying a function to find matches and another function to update them:

db.update(
    record => record.name === "James",
    record => { record.nickname = "Bulldog"; return record; }
).then(results => /* do something */ );

Delete some records from the database by supplying a function to find matches:

db.delete(
    record => record.modified < Date.now()
).then(results => /* do something */ );

Delete the database:

db.drop().then(results => /* do something */ );

Constructor

Creates a new instance of an NJODB Database.

Parameters:

If an njodb.properties file already exists in the root directory, a connection to the existing Database will be created. If the root directory does not exist it will be created. If no user-specific properties are supplied, an njodb.properties file will be created using default values; otherwise, the user-supplied properties will be merged with the default values (see Database properties below). If the data and temp directories do not exist, they will be created.

Example:

const db = new njodb.Database() // created in or connected to the current directory

const db = new njodb.Database("/path/to/some/other/place", {datadir: "mydata", datastores: 2}) // created or connected to elsewhere with user-supplied properties

Database properties

An NJODB Database has several properties that control its functioning. These properties can be set explicitly in the njodb.properties file in the root directory; otherwise, default properties will be used. For a newly created Database, an njodb.properties file will be created using default values.

Properties:

Database management methods

stats

stats

Returns statistics about the Database. Resolves with the following information:

statsSync

A synchronous version of stats.

grow

grow()

Increases the number of datastores by one and redistributes the data across them.

growSync

growSync()

A synchronous version of grow.

shrink

shrink()

Decreases the number of datastores by one and redistributes the data across them. If the current number of datastores is one, calling shrink() will throw an error.

shrinkSync

shrinkSync()

A synchronous version of shrink.

resize

resize(size)

Changes the number of datastores and redistributes the data across them.

Parameters:

resizeSync

resizeSync(size)

A synchronous version of resize.

drop

drop()

Deletes the database, including the data and temp directories, and the properties file.

dropSync

dropSync()

A synchronous version of drop.

getProperties

getProperties()

Returns the properties set for the Database. Will likely be deprecated in a future version.

setProperties

setProperties(properties)

Sets the properties for the the Database. Will likely be deprecated in a future version.

Parameters:

Data manipulation methods

insert

insert(data)

Inserts data into the Database.

Parameters:

Resolves with an object containing results from the insert:

insertSync

insertSync(data)

A synchronous version of insert.

insertFile

insertFile(file)

Inserts data into the database from a file containing JSON data. The file itself does not need to be a valid JSON object, rather it should contain a single stringified JSON object per line. Blank lines are ignored and problematic data is collected in an errors array.

Resolves with an object containing results from the insertFile:

An example data file, data.json, is included in the test subdirectory. Among many valid records, it also includes blank lines and a malformed JSON object. To insert its data into the database:

db.insertFile("./test/data.json").then(results => /* do something */ );

insertFileSync

insertFileSync(file)

A synchronous version of insertFile.

select

select(selecter [, projector])

Selects data from the Database.

Parameters:

Resolves with an object containing results from the select:

Example with projection that selects all records, returns only the id and modified fields, but also creates a new one called newID:

db.select(
    () => true,
    record => { return {id: record.id, newID: record.id + 1, modified: record.modified }; }
);

selectSync

selectSync(selecter [, projector])

A synchronous version of select.

update

update(selecter, updater)

Updates data in the Database.

Parameters:

Resolves with an object containing results from the update:

updateSync

updateSync(selecter, updater)

A synchronous version of update

delete

delete(selecter)

Deletes data from the Database.

Parameters:

Resolves with an object containing results from the delete:

deleteSync

deleteSync(selecter)

A synchronous version of delete.

aggregate

aggregate(selecter, indexer [, projecter])

Aggregates data in the database.

Parameters:

Resolves with an object containing results from the aggregate:

Each index object contains the following:

Each aggregation object contains one or more of the following (non-numeric fields do not contain numeric aggregate data):

An example that generates aggregates for all records and fields, grouped by state and lastName:

db.aggregate(
    () => true,
    record => [record.state, record.lastName]
);

Another example that generates aggregates for records with an ID less than 1000, grouped by state, but for only two fields (note the non-numeric fields do not include numeric aggregate data):

db.aggregate(
    record => record.id < 1000,
    record => record.state,
    record => { return {favoriteNumber: record.favoriteNumber, firstName: record.firstName}; }
);

Example aggregate data array:

[
    {
        index: "Maryland",
        count: 50,
        aggregates: [
            {
                field: "favoriteNumber",
                data: {
                      min: 0,
                      max: 98,
                      sum: 2450,
                      mean: 49,
                      varp: 833,
                      vars: 850,
                      stdp: 28.861739379323623,
                      stds: 29.154759474226502
                  }
            },
            {
                field: "firstName",
                data: {
                    min: "Elizabeth",
                    max: "William"
                }
            }
        ]
    },
    {
        index: "Virginia",
        count: 50,
        aggregates: [
            {
                field: "favoriteNumber",
                data: {
                    min: 0,
                    max: 49,
                    sum: 1225,
                    mean: 24.5,
                    varp: 208.25000000000003,
                    vars: 212.50000000000003,
                    stdp: 14.430869689661813,
                    stds: 14.577379737113253
                }
            },
            {
                field: "firstName",
                data: {
                    min: "James",
                    max: "Robert"
                }
            }
        ]
    }
]

aggregateSync

aggregate(selecter, indexer [, projecter])

A synchronous version of aggregate.

Finding and fixing problematic data

Many methods return information about problematic records encountered (e.g., records that are not parseable using JSON.parse(), or ones that couldn't be updated or deleted); both a count of them, as well as details about them in the details array. The objects in the details array - one for each datastore - contain an errors array that is a collection of objects about problematic records.

For un-parseable records, each error object includes the line of the datastore file where the problematic record was found as well as a copy of the record itself. With this information, if one wants to address these problematic data they can simply load the datastore file in a text editor and either correct the record or remove it. For records that couldn't be deleted or updated, each error object includes a copy of the record itself. With this information, one could make another attempt to update or delete the record(s), or otherwise handle the failure.

Here is an example of the details for a datastore that contains an un-parseable record. As you can see, the record is on the tenth line of the file, and the problem is that the lastname key name is missing an enclosing quote. Simply adding the quote fixes the record.

{
  store: '/Users/jamesbontempo/github/njodb/data/data.0.json',
  size: 1512464,
  lines: 8711,
  records: 8709,
  errors: [
    {
      error: 'Unexpected token D in JSON at position 42',
      line: 10,
      data: '{"id":232,"firstName":"Robert","lastName:"Davis","state":"Illinois","birthdate":"1990-10-22","favoriteNumbers":[5,34,1],"favoriteNumber":183,"modified":1616806973645}'
    }
  ],
  blanks: 1,
  created: 2021-03-27T01:20:21.562Z,
  modified: 2021-03-27T01:28:32.686Z,
  start: 1616808517081,
  end: 1616808517124,
  elapsed: 43
}