flexnes v0.8.0

Table of contents

• Getting started
  • Start the server using the pre-built official Docker images
  • CRUD examples
  • Stop the server
  • Build and run the server as Docker containers
  • Flexnes test suite for local testing
• REST API example server
  • NoSql API quick overview
    • NoSql Query language
  • Key-value API quick overview
  • Security
• Standalone gRPC Datastore server
  • Start the gRPC server
• Example Javascript HTTP client
• Example Node.js gRPC client

Flexnes

Flexnes (flexible network store) is a database engine for modern web apps and middle-tier backend services. It's built with Google's LevelDB, gRPC and Node.js. Flexnes is also used as an open core in the Serverless SaaS codehooks.io.

Flexnes supports query languages like NoSql (Mongodb like) and Key-Value (Redis like), which makes it a good candidate for persitence storage in your JAMStack, middle-tier and micro-service programming.

Flexnes implements two main APIs:

• Clean REST API for all-purpouse programming
• Low level gRPC API for high performance and real time computing.

This repository har two main parts:

• The Datastore engine source code: ./src/dbengine
• The REST API HTTP server source code: ./src/restapi.

Getting started

Requirements: Docker and docker-compose.

Start the server using the pre-built official Docker images

# start Flexnes server and REST API server
docker-compose -f official.yml up -d

For local development and changes to the source code: install all packages locally and start the server using the node.js and npm

# clone repo first for local development
git clone https://github.com/RestDB/flexnes.git
cd flexnes
# install dependencies
npm install
# generate development certs
npm run cert
# start REST API server and gRPC server
npm run flexnes

CRUD examples

Add some data with curl

curl --location --request POST 'http://127.0.0.1:3000/dev/data/mycoll' \
--header 'x-apikey: daejoo' \
--header 'Content-Type: application/json' \
--data-raw '{
    "hello": "world!"
}'

# output example
{"hello":"world!","_id":"60ec427298586b1e00000001"}

Get some data with curl

curl --location -g --request GET 'http://127.0.0.1:3000/dev/data/mycoll' \
-G -d 'q={"hello":{"$eq":"world!"}}' \
--header 'x-apikey: daejoo' \
--header 'Content-Type: application/json'

# output example (always array of data from a query)
[{"hello":"world!","_id":"60e86150c9e5731800000001"}]

Update some new data

curl --location --request PUT 'http://127.0.0.1:3000/dev/data/mycoll/60ee9924359a3d1800000001' \
--header 'x-apikey: daejoo' \
--header 'Content-Type: application/json' \
--data-raw '{
    "hello": "universe!"
}'

Delete some data

curl --location --request DELETE 'http://127.0.0.1:3000/dev/data/mycoll/60ee9924359a3d1800000001' \
--header 'x-apikey: daejoo' \
--header 'Content-Type: application/json' \
--data-raw '{
    "hello": "universe!"
}'

Stop the server

docker-compose -f official.yml down

Build and run the server as Docker containers

# build all docker images
docker-compose build

# start Flexnes server and REST API server
docker-compose up -d

# check logs
docker-compose logs -f

# stop all servers
docker-compose down

Flexnes test suite for local testing

Run these if you modify the source code. The Mocha test suite has 54 unit tests and they are located here: src/dbengine/test/test.js.

To run the tests Node.js is required.

# install packages
npm install

# run tests
npm run test-server

A succesfull test run should show the a similar output:

 Flexnes datastore engine tests
    ✓ should connect to server
    ✓ should drop a collection
    ✓ should drop huge collection
    ✓ should create test collections and data
    ...
    ✓ should restore database


  56 passing (13s)

REST API example server

Flexnes comes with a simple Node express REST API server. The following APIs are avaliable.

NoSql API quick overview

URL:

HTTP VERB /{datastore}/data/{collection}?[query params]

NoSql Query language

Flexnes uses Mongodb query language as implemented by Sift.

E.g.

GET http://localhost:3000/dev/data/places?q={ $in: ["Costa Rica", "Brazil"] }

Key-value API quick overview

All API calls to the key-value Datastore are done as POST with a payload body that describes the command.

URL:

POST /{datastore}/kvdata

E.g.

POST http://localhost:3000/dev/kvdata

{
    "set": {"key": "temp", "value": "hotter"},
    "opt": {"ttl": 60000}
}

Security

The REST API server comes with a simple config file that provides the various API tokens for accessing different data stores. The default config file is located here: ./src/restapi/config.json. Override the config by mouting /www/restapi/config.json to local file you provide.

E.g. the example config.json file located in the root folder lists two API tokens, one for the dev data store and another for the test data store.

{
    "apikeys": {
        "dev": {
            "tjobing": "only for development"
        },
        "test": {
            "pwd123": "only for testing"
        }
    },
    "ip-whitelist": {
        "127.0.0.1": "localhost"
    }
}

Standalone gRPC Datastore server

For high performance applications it is possible to run Flexnes as a gRPC server only, exposing the Protobuf API via gRPC. This allows for any programming language that supports the gRPC protocol to communicate with the Datastore server.

The gRPC API is shown below. You will find the code under src/dbengine/protos/fxs.proto

// The Flexnes service definition.
service Fxs {
  // session
  rpc login (LoginRequest) returns (Response) {}
  
  // nosql db api
  rpc save (SaveRequest) returns (Response) {}
  rpc get (QueryRequest) returns (stream Response) {}
  rpc ensureIndex (IndexRequest) returns (stream Response) {}
  rpc rebuildIndex (IndexRequest) returns (Response) {}
  rpc removeIndex (IndexRequest) returns (Response) {}
  rpc count (CountRequest) returns (Response) {}
  rpc update (UpdateRequest) returns (Response) {}
  rpc dropCollection (DropCollectionRequest) returns (Response) {}
  rpc delete (DeleteRequest) returns (Response) {}
  rpc getKeys (KeysRequest) returns (stream Response) {}
  rpc getValues (ValuesRequest) returns (stream Response) {}
  
  // kv db api
  rpc kvset(KVSetRequest) returns (Response) {}
  rpc kvget(KVGetRequest) returns (Response) {}
  rpc kvdelete(KVDeleteRequest) returns (Response) {}
  rpc kvgetall(KVGetRequest) returns (stream Response) {}
  rpc kvdeleteall(KVDeleteRequest) returns (Response) {}
  rpc kvincr(KVIncrRequest) returns (Response) {}
  rpc kvdecr(KVIncrRequest) returns (Response) {}
  rpc getChangelog(ChangelogRequest) returns (stream Response) {}
  
  // backup restore api
  rpc backup(Empty) returns (stream BackupRequest) {}
  rpc restore(stream BackupRequest) returns (Response) {}
  
  // utility api
  rpc ping(Empty) returns (stream Response){}
  rpc realtime(RealTimeRequest) returns (stream Response){}
  rpc getDBSize(Empty) returns (Response){}
  rpc createDatabase(CreateDatabaseRequest) returns (Response){}
}

Start the gRPC server

Starts the server at the default port 50051

docker-compose -f flexnes-dev.yml up

Example Javascript HTTP client

...

Example Node.js gRPC client

...