Valenciadb-cs NPM

valenciadb-cs (ValenciaDB Community Server)

ValenciaDB is a NoSQL database mangement system. Concretely, a key-value store.

Features:

It supports in-memory and on-disk stores.
It allows to use schemaless stores and static schema stores.
It supports on-disk views aka materialized views.
It allows to set the key field. Unlike other DBMS, ValenciaDB allows to select our own key field.
It supports auto key fields. ValenciaDB allows to use implicit and explicit key values.
It supports primary and secondary indexes.
It supports BEFORE and AFTER triggers written in JavaScript.
It supports time-to-live documents. We can insert documents with a time-to-live. When this time expires, the server removes the document automatically.
It supports fixed-size stores. We can set the maximum number of documents to contain.
It supports role-based security and audit.
It supports read-only/static stores.
It provides a shell for running queries.
It provides a JavaScript driver for running queries from Node.js and the browsers.
It can be accessed from the browser with CORS requests.
Easy to learn and easy to use.
It uses JSON for storing the data.

Installing
Database instance
Drivers
Stores
Key-value stores
Dropping stores
System stores
Security
Time-to-live
Fixed-size stores
Triggers
Static stores
Views
Static schemas
Secondary indexes

Installing

We install ValenciaDB using npm:

npm install -g valenciadb-cs valenciash

valenciadb-cs installs ValenciaDB Community Server. And valenciash, the shell. The following commands will be available:

valenciadb to create, start and stop database instances. Installed by valenciadb-cs.
valenciash to connect a database instance using the shell. Installed by valenciash.
valenciaexp to export data. Installed by valenciash.
valenciaimp to import data. Installed by valenciash.

Database instance

A database instance is a DBMS in running.

Creating a database instance

You can create a new database instance using the command valenciadb:

valenciadb create <folder>
valenciadb create <folder> --host <name>

Through the host option, we can set the server name in the configuration file. By default: localhost.

Example:

$ valenciadb create db
Database instance created.
$ ll db
total 20
drwxrwxr-x 4 me me 4096 feb 10 13:54 ./
drwxrwxr-x 3 me me 4096 feb 10 13:54 ../
drwxrwxr-x 4 me me 4096 feb 10 16:58 data/
drwxrwxr-x 2 me me 4096 feb 10 16:37 logs/
-rw-rw-r-- 1 me me  463 feb 10 13:54 valenciadb.conf.json
$

valenciadb.conf.json

The configuration file is valenciadb.conf.json. This is created by the valenciadb create command. We can modify it, but no property must be removed.

Starting a database instance

An instance serves one database. You can start a server using the command valenciadb:

valenciadb start
valenciadb start confFile

When the first syntax is used, the configuration file must be in the current directory.

By default, the instance binds http://localhost:46000.

Drivers

Right now, ValenciaDB provides one driver, the JavaScript driver. This can be used from Node.js and the browser. Coming soon, ValenciaDB is going to support other drivers, for Go and Lua..

Browser driver

If you need to access to the database instance from the browser, you can get the driver from the own instance:

<script src="http://localhost:46000/_vdb/driver/javascript/browser.js"></script>

You will have the global valenciadb object.

Node.js driver

To use the driver in Node.js:

const valenciadb = require("valenciadb-driver");

Connecting using the JavaScript driver

To connect to a database instance, we must do the following:

const driver = new valenciadb.Driver();
const cx = driver.createConnection();
var db;

cx.open({
  endpoint: "http://localhost:46000",
  username: "root",
  password: "valencia"
}).then(
  () => {
    console.log("Connected");
    db = cx.db;
  },

  (err) => console.dir(err)
);

Stores

A store is a document container, where a document is the formal name for a record.

Key-value stores

A key-value store is a kind of store where every document is identified by a special field, known as key field. The documents are always accessed by the key field. No other field possible. We can use any field as key, that must be specified when the store is created.

The key-value stores are used by many organizations, for example, Amazon, BBC, bet365, Cisco, Docker, eBay, Facebook, General Electric, Google, IBM, Kayak, LinkedIn, Microsoft, Mozilla Foundation, NASA, The National Health Service UK, PayPal, Pivotal, Ryanair, Sky, Symantec, Uber, Verizon and Virgin.

The main use cases are:

Data cache.
Session data.
Reporting.
Product catalog.
Logging.
Messaging.

Reading stores

To read data, you can use the JavaScript driver or the shell.

To start the shell:

$ valenciash -e http://localhost:46000 -u root -p valencia
Connecting as root at http://localhost:46000...
root@http://localhost:46000>

The valenciash documentation is available on https://www.npmjs.com/package/valenciash.

Getting the number of documents

To get the number of documents a store has:

//shell
db.store.count()

//JavaScript driver
db.store.count().then(
  (count) => ...,
  (err) => ...
)

Checking whether a key exists

To check whether a key is in use:

//shell
db.store.exists({keyField: value})

//JavaScript driver
db.store.exists({keyField: value}).then(
  (exists) => ...,
  (err) => ...
)

Example:

> db.bands.exists({pk: "echo-and-the-bunnymen"})
true
>

Getting documents

To get documents:

//shell
db.store.find(filter)     //return documents in an array
db.store.findAll()        //return all documents in an array
db.store.findOne(filter)  //return one document

//JavaScript driver
db.store.find(filter).then(
  (docs) => ...,
  (err) => ...
)

db.store.findAll().then(
  (docs) => ...,
  (err) => ...
)

db.store.findOne(filter).then(
  (doc) => ...,
  (err) => ...
)

The filter is an object containing the condition. In a key-value store, the filter must be on the key field. The filter has the following syntax:

{field: value}        //similar to: field == value
{field: {$op: value}} //similar to: field op value

Example:

db.bands.find({pk: "athlete")
db.bands.find({pk: {$in: ["athlete", "the-church"]}})

The operators are:

$eq: Equality. Example: {pk: {$eq: "athlete"}}.
$ne: Inequality. Example: {pk: {$ne: value}}.
$lt: Less than. Example: {pk: {$lt: "b"}}.
$le: Less than or equal to. Example: {pk: {$le: "b"}}.
$gt: Greater than. Example: {pk: {$gt: "b"}}.
$ge: Greater than or equal to. Example: {pk: {$ge: "b"}}.
$between: In a range. Example: {$pk: {$between: ["a", "h"]}}.
$notBetween or $notbetween: Example: {$pk: {$notbetween: ["a", "h"]}}.
$in. Into a array. Example: {pk: {$in: ["athlete", "the-church"]}}. . $notIn or $notin. Example: {pk: {$notin: ["athlete", "the-church"]}}
$like. Like pattern. Example: {pk: {$like: /the-%/}}.
$notLike or $notlike. Example: {pk: {$notlike: /the-%/}}.

Key-value queries

The key-value stores provide the q() method to perform complexer queries. This method returns an instance of KeyValueQuery, which provides the following methods:

filter(filter) : KeyValueQuery. It sets the filter. It returns the query for chaining other methods. We only can filter using the key field or an indexed field.
orderBy(sort) : KeyValueQuery. It sets how to order the result.
limit(max) : KeyValueQuery. It sets the maximum number of documents to return the query.
offset(skip) : KeyValueQuer. It sets the first documents to skip.
exists() : Promise. It runs the query checking whether some document passes the filter.
findOne() : Promise. It runs the query and returns the first document from the result.
find() : Promise. It runs the query and returns an array with the result.

Examples:

db.bands.q().filter({name: {$like: "The %"}}).find(
  (res) => ...,
  (err) => ...
)

db.bands.q().filter({name: {$like: "The %"}}).orderBy({name: "asc"}).limit(10).find(
  (res) => ...,
  (err) => ...
)

Let's remember the following methods of the key-value stores:

db.store.find(filter) similar to db.store.q().filter(filter).find()
db.store.findOne(filter) similar to db.store.q().filter(filter).findOne()
db.store.exists(filter) similar to db.store.q().filter(filter).exists()

Additionally, we also have:

db.store.filter(filter) similar to db.store.q().filter(filter)

Inserting documents

To insert a new document:

//shell
db.store.insert(doc)

//JavaScript driver
db.store.insert(doc).then(
  () => ...,
  (err) => ...
)

Example:

> db.bands.insert({pk: "the-smiths", name: "The Smiths", origin: "UK"})

Replacing a document

To replace a document:

//shell
db.store.replace(doc)

//JavaScript driver
db.store.replace(doc).then(
  () => ...,
  (err) => ...
)

The document to replace is that has the key field indicated in the document. Example:

> db.bands.insert({pk: "nada-surf", name: "Nada Surf", origin: "US"})
> db.bands.findOne({pk: "nada-surf"})
{ pk: 'nada-surf', name: 'Nada Surf', origin: 'US' }
> db.bands.replace({pk: "nada-surf", name: "NADA SURF"})
> db.bands.findOne({pk: "nada-surf"})
{ pk: 'nada-surf', name: 'NADA SURF' }
>

Updating a document

To update a document:

//shell
db.store.update(filter, setter)

//JavaScript driver
db.store.update(filter, setter).then(
  () => ...,
  (err) => ...
)

The setter is an object to contain the update. The syntax for each field is as follows:

field: value
field: {$operator: value}

Example:

> db.bands.findOne({pk: "the-house-of-love"})
{ pk: 'the-house-of-love', name: 'The House of Love' }
> db.bands.update({pk: "the-house-of-love"}, {origin: "UK"})
> db.bands.findOne({pk: "the-house-of-love"})
{ pk: 'the-house-of-love',
  name: 'The House of Love',
  origin: 'UK' }
>

The operators are:

$del. Remove a field from the document. Example: tags: {$del: true}. If $del is false, no op performed.
$set. Set the value or create the field into the document. Example: age: {$set: 41}.
$inc. Increment the value. Example: age: {$age: 3}.
$dec. Decrement the value. Example: age: {$dec: 3}.
$mul. Multiply the value. Example: level: {$mul: 2}.
$add. Add an item to a set. Example: tags: {$add: "new-tag"}.
$push or $rpush. Add an item at the end of an array. Example: tasks: {$push: "new-task"}.
$pop or $rpop. Remove the last items of an aray. Example: tags: {$pop: 1}.
$unshift or $lpush. Add an item at the start of an array. Example: tasks: {$unshift: "new-task"}.
$shift or $lpop. Remove the first items of an aray. Example: tags: {$shift: 1}.
$remove or $rem. Remove an item of an array. tags: {$remove: "indie"}.
$concat. Concatenate a string at the end of other or an array at the end of another. Example: message: {$concat: "hello"}.

The operators $push, $pop, $unshift and $shift are similar to the JavaScript Array methods. For the Redis fans, we have created the aliases $lpush, $lpop, $rpush, $rpop and $rem.

Removing documents

To remove document(s):

//shell
db.store.remove(filter)

//JavaScript driver
db.store.remove(filter).then(
  () => ...,
  (err) => ...
)

To remove all documents:

//shell
db.store.truncate()

//JavaScript driver
db.store.truncate().then(
  () => ...,
  (err) => ...
)

Creating key-value stores

The creation of a key-value store is very simple. Unlike other database engines, no special command provided. We have to insert a new document into the db.stores store. Simple.

Example:

//JavaScript driver
db.stores.insert({
  name: "bands",        //store name
  type: "kv",           //store type: kv for key-value stores
  engine: "on-disk",    //where the store is saved: on-disk or in-memory
  key: "pk"             //key field name
}).then(
  () => ...,
  (err) => ...
)

Once you have created a store, you can access it using db.storeName. For example: db.bands.

An in-memory store is automatically created every time the instance is started. Its access is faster than on-disk stores, but its content is truncated when the instance is stopped formal or informally.

Keys

Every document has a key, the field to identify it from the rest. The primary key of the documents. We must indicate it in the key field when the store is created:

//a way
key: "field-name"

//other one
key: {
  field: "field-name"
}

We can use explicit or implicit keys. When an explicit key used, the key value must be set by the user explicitly. It's the default behavior. Instead, if an implicit key used, the key can be set by the instance when the user doesn't set it.

To use implicit keys, we can define the store with a key field as follows:

key: {
  field: "field-name",
  value: "uuid"         //UUID v1 will be used when not set by the user
}

Dropping stores

To drop a store, you have to remove its document from db.stores. Example:

> db.stores.remove({name: "bands"})

System stores

A system store is created by ValenciaDB. Right now, the built-in stores are:

db.audit. Documents generated by the auditor.
db.logs. Instance event/message log.
db.roles. Security roles.
db.sessions. Current user sessions.
db.stores. Stores created in the instance.
db.users. Users.

In the future:

db.db
db.params

Security

Role-based security

ValenciaDB uses a role-based security. A role is a set of privileges that can be granted to the users. The roles are inserted into db.roles. The users into db.users.

Role document:

name (string). Role name. The key field.
system (bolean). Is a system role?
desc (string). Role description.
extra (object). Additional/extra data about the role.
active (boolean). Is it active?
privileges (string[]). Privileges granted to the role.

Example:

> db.roles.findOne({name: "db.$.*"})
{ name: 'db.$.*',
  system: true,
  desc: 'Allow all on system stores.',
  extra: {},
  active: true,
  privileges:
   [ 'db.$.find',
     'db.$.insert',
     'db.$.replace',
     'db.$.update',
     'db.$.remove',
     'db.$.truncate' ] }
>

A privilege is a right to do something. Its syntax is as follows:

db.storeName.method

db must always be specified as db.

The store name can be: a store name; * to indicate all user stores; or $ to set all system stores.

The method can be: *, find, insert, replace, remove o truncate.

Some examples:

db.$.*          //granted all on system stores
db.*.*          //granted all on user stores
db.$.find       //granted find on system stores
db.*.insert     //granted insert on user stores
db.users.*      //granted all on db.users
db.roles.insert //granted insert on db.roles

User document:

name (string). Username.
password (string). User password.
desc (string). User description.
extra (object). Additional/extra user data.
active (boolean). Is it active?
roles (string[]). Roles granted to the user.

Example:

> db.users.findOne({name: "root"})
{ name: 'root',
  password: 'valencia',
  desc: 'Superuser.',
  extra: {},
  active: true,
  roles: [ 'db.$.*', 'db.*.*' ] }
>

To create a new user, insert a new document into db.users:

> db.users.insert({name: "me", password: "my-pwd", desc: "My account.", roles: ["db.$.*", "db.*.*"]})

To change a password, update the user document:

> db.users.update({name: "me"}, {password: "new-password"})

The current user sessions are saved into the db.sessions store. If you need to close a session, remove its document from db.sessions.

Audit

The audit is the mechanism to monitor the access to the stores. It is enabled in the valenciadb.conf.json file, using the security.audit property:

enabled (boolean). Is the audit enabled?
events (object). Events to audit. Right now, we can set the store event whose value is an array similar to the privileges.

Example:

"security": {
  ...

  "audit": {
    "enabled": true,
    "events": {
      "store": [
        "db.$.*",
        "db.*.*"
      ]
    }
  }
}

The db.audit store contains the audit documents.

Time-to-live

ValenciaDB provides time-to-live stores. A TTL store can remove documents when their TTLs expire. For example, the db.sessions is a TTL store removing the sessions when 30 minutes have passed without using. Example:

root@http://localhost:46000> db.stores.findOne({name: "sessions"})
{ id: 13,
  name: 'sessions',
  type: 'kv',
  storage: 'in-memory',
  system: true,
  key: { field: 'key', value: null },
  indexes: null,
  schema:
   { key: { type: 'string', required: true },
     username: { type: 'string', required: true },
     privileges: { type: 'array', required: true },
     start: { type: 'string', required: true } },
  desc: 'Sessions.',
  ttl: { enabled: true, default: '30m' },
  mutable: true,
  triggers:
   { before:
      { when: null,
        fn: '/*eslint no-unused-vars: "off"*/\nfunction before(event, old, nw) {\n  if (["insert", "replace", "truncate"].includes(event)) {\n    throw new Error(`${event}() unsupported on db.sessions.`);\n  } else if (event == "update") {\n    if (old.username != nw.username ||\n        old.privileges != nw.privileges ||\n        old.key != nw.key ||\n        old.start != nw.start) {\n      throw new TypeError("Only ttl field can be updated.");\n    }\n\n    return nw;\n  }\n}\n' },
     after: null },
  size: { max: null } }
root@http://localhost:46000>

A TTL store is created using the ttl field. An object containing:

enabled (boolean). Is a TTL store?
default (string). Default TTL. Example: 1h. Suffixes: s, seconds; m, minutes; h, hours; d, days; w, weeks; M, months; and y, years.

When a document is inserted into a TTL store, the default TTL is set. We can indicate other one, using the ttl field. Example:

db.logs.insert({
  ttl: "1h", //1h from now
  ...
})

To touch the document TTL, update it:

db.tasks.update(
  {id: "1234"},
  {ttl: "30m" /*30m more from now*/}
)

Fixed-size stores

A fixed-size store can contain a maximum number of documents. This is set with the size field of the store document. Example:

> db.stores.findOne({name: "applogs"})
{ id: 120,
  key: 'key',
  name: 'applogs',
  type: 'kv',
  storage: 'on-disk',
  system: false,
  desc: 'App logs.',
  ttl: { enabled: false, default: null },
  mutable: true,
  triggers: { before: null, after: null },
  size: { max: 10000 } }
>

Triggers

A trigger is a JavaScript function that must be executed before or after every document-level operation (insert, replace, update or remove) or command-level operation (truncate). The triggers are set using the triggers field of the store documents. The persistent stores can have one BEFORE trigger and other AFTER trigger.

Creating a trigger:

> db.stores.update(
  {
    name: "mystore"
  },
  {
    triggers: {
      before: String(function(event, nw, old) { ... }),
      after: String(function(event, nw, old) { ... })
    }
  }
)

Important. The trigger functions must be specified as strings.

The trigger functions can define the following parameters:

db (object). Database object. Only AFTER triggers.
nw (object). When insert, it contains the new document; when update or replace, the new document to store.
old (object). When update or replace, the original document before modifying. If remove, the document to remove.
event (string). The operation: insert, replace, update, remove or truncate.
username (string). User name.
done (function). Function to notify the call end when asynchronous. Only AFTER triggers.

The parameters aren't positional. They are nominal. We have to use the given names, in any order.

We can cancel an operation throwing an exception in a BEFORE trigger.

When a BEFORE trigger defined, it has to return the document to insert or to update. Similar to the PostgreSQL way.

Restriction

The triggers can define a condition to select when it must be called, similar to MariaDB, PoastgreSQL and SQLite. This condition is defined as a string using the when property and the trigger function using func. Example:

> db.stores.update(
  {
    name: "mystore"
  },
  {
    triggers: {
      before: {
        when: "event == 'insert'",
        func: String(function(event, nw, old) { ... })
      },

      after: String(function(event, nw, old) { ... })
    }
  }
)

Asynchronous triggers

The BEFORE triggers are always synchronous, but the AFTER triggers can be synchronous or asynchronous. The default type is synchronous. An asynchronous trigger is created using the done parameter, used by the trigger to notify that has ended. Example:

after: String(function(event, done) {
  //perform asynchronous something

  done(); //end
})

done parameter signatures:

done()
done(error)

Important aspects about triggers

The BEFORE trigger is always synchronous.
The AFTER trigger can be synchronous or asynchronous.
To cancel an operation, this only can be done by the BEFORE trigger.
If the AFTER trigger fails, the operation finishes OK. Please, remember: the AFTER trigger never cancels or aborts the query.
The db and done parameters are available into an AFTER trigger, never into BEFORE one.
The trigger function code must be passed as a string.

Removing a trigger

To remove a trigger, set the null value.

Static stores

A static store is a read-only store. It's set with the mutable field. If false, it is read-only store; otherwise, read/write.

Views

A view is a store whose documents are from other store. The database engine maintains its documents. Similar to the materialized views of SQL.

Example for creating a view:

db.stores.insert({
  name: "indies",
  type: "kv",
  key: "pk",
  storage: "on-disk",
  view: {
    for: "bands",
    map: String(function(doc) {
      if (doc.tags.includes("indie")) {
        return {
          pk: doc.pk,
          name: doc.name
        };
      }
    })
  }
}).then(
  () => ...,
  (err) => ...
)

The map function indicates the documents that must be saved into the view for a given document:

If a document not a part of the view, the function must return nothing.
If a document generates a view document (1-to-1 relationship), the function must return the document view.
If a document generates several view documents (1-to-* relationship), the function must return an array with the document views.

Example returning several documents:

db.stores.insert({
  name: "dict",
  type: "KV",
  key: "pk",
  storage: "on-disk",
  view: {
    for: "words",
    map: String(function(doc) {
      const res = [];
      for (let word of doc.words) res.push({pk: word, src: doc.pk});
      return res;
    })
  }
}).then(
    () => ...,
    (err) => ...
)

Important:

Two or more view documents can never have the same value for the key field.
The map function must be passed as string.

Static schemas

By default, the key-value stores are schemaless, that is, the database engine doesn't validate the document schemas. If needed, we can define the schema for a store, using the schema property.

Exemple:

> db.stores.insert({
  name: "bands",
  type: "kv",
  storage: "on-disk",
  key: "pk",
  schema: {
    pk: "string",
    name: "string",
    origin: "string",
    year: "number"
  }
})

The schema field is an object where the properties represent fields whose values contain the definitions. These definitions can be a string to indicate the data type or an object as follows:

type (string). Data type: any, array, boolean, number, string and object.
required (boolean). Is a field required? By default: true.
check (array). Possible values if string, number or any.
default (any). Default value.

Other example:

> db.stores.insert({
  name: "bands",
  type: "kv",
  storage: "on-disk",
  key: "pk",
  schema: {
    pk: {type: "string"},
    name: {type: "string"},
    origin: {type: "string", required: false},
    year: {type: "number", required: false},
    genre: {type: "string", required: false, check: ["indie", "alt", "pop", "rock"]}
  }
})

Secondary indexes

The key-value stores only can be queried by the key field. If we need to query using another field, we have to define a secondary index for that field.

A primary index is one for the key field. This is created automatically by the instance. A secondary field is one for a non-key field to allow the queries for that field.

Creating a secondary index

The indexed fields are defined through the indexes property. Example:

> db.stores.insert({
  name: "bands",
  type: "kv",
  storage: "on-disk",
  key: "pk",
  indexes: ["name", "origin"]
})

The indexes field is an array where each item represents an index for the field name indicated. The items can be a string, that is, the field name to index, or an object as follows:

fields (string[]). The field names to index. Right now, the indexes are simple, that is, we only can indicate a field.
unique (boolean). Is it a unique index? Default: false.

Let's see another example:

> db.stores.insert({
  name: "bands",
  type: "kv",
  storage: "on-disk",
  key: "pk",
  indexes: [{fields: ["name"], unique: true}, {fields: ["origin"]}]
})

The method filter() allows only one field: the key field or an indexed field. Please, don't forget it! This is a characteristic of the key-value stores.

NoSQL database NoSQL database Key-value store Key-value database

async body-parser cookie-parser express valenciadb-lib yargs

7 years ago

7 years ago

7 years ago

7 years ago

7 years ago