bfast-database v4.0.0-beta.2

bfast-database

NodeJS Database Runtime Instance For Bfast::Cloud::Database.

Get Started For Linux

This instruction tested for linux environment you can translate the commands to your OS.

• You need to have NodeJs to your local machine, or you can download from NodeJs website.

• You need to have mongodb in your local machine or mongodb url. To install mongodb go to mongodb installation

Install BFast Tools

To run this database instance you must have bfast-tools package in your local

computer to install run $ sudo npm install -g bfast-tools

Pull tarball file of bfast-database from npm

Open your terminal into a folder you want to run database instance then execute the folloeing

$ npm pack bfast-database

Command will download packed bfast-database in .tgz format

Unpack bfast-database

Unpack the downloaded file $ tar -xf bfast-database-**.tgz. Note ** mean any latest version number fill will be tagged to.

Run bfast-database

After unpack you bfast-database files will be inside package folder.

Start in restart mode

    $ cd package
    $ bfast functions serve  --port 3000 

Start in non - restart mode

    $ cd package
    $ bfast functions serve --static --mongodb-url="${MONGO_URL}"  --port 3000

Then your bfast-database instance is up and running.

Configurations

You put all configurations in ENV if you want a node module version please see bfast-database-core

Mandatory envs

  - APPLICATION_ID=bfast // supply appId the default or when PRODUCTION === '0' is [ bfast ]
  - MASTER_KEY=bfast // your master key default or when PRODUCTION === '0' is [ bfast ]
  - PROJECT_ID=bfast // your projectId default or when PRODUCTION === '0' is [ bfast ] 
  - DATABASE_URI="mongodb://localhost/test"// mongodb url to save your data, if in development mode and you don't supply it default is [ 'mongodb://localhost/bfast' ]
  - PORT=3000 // port to serve bfast-database if you use bfast-tools it will pick the port you specify with --port or 3000 as default
  - RSA_KEY='{..}' // stringfy JSON of your rsa private key for signed jwt
  - RSA_PUBLIC_KEY='{..}' // stringfy of your rsa public key for signed jwt

Optional envs

  - PRODUCTION="0" // specify whether you run in production or developement mode when '0' is dev and '1' is production, default value is [ '0' ]

Storage if you want to use amazon s3 object storage, you must supply its configurations as follows. If not bfast-database will use gridfs api of mongodb

  - S3_BUCKET= // you s3 bucket name for aws storage 
  - S3_ACCESS_KEY= // you s3 access key for aws storage 
  - S3_SECRET_KEY= // your s3 secret key
  - S3_REGION=af-south-1 // your s3 region
  - S3_ENDPOINT=https://s3.af-south-1.amazonaws.com // your s3 endponit

Storage if you want to use GridFs when do not supply any of s3 configuration when you start bfast-database, and it will opt to use GridFs.

Example For Linux Users

Pull the repository git clone https://github.com/fahamutech/bfast-database.

Then run start.js file, just replace the env variable of your choice.

    $ node start.js

That's it. For documentation refer to bfast-database-core repository

Documentations

BfastDatabase Engine

Database as a service application to work with mongoDb as a primary or any other database.

1.0 Introduction

Backend application written to act upon a database to respond to user actions on the client side. Those services exposed to REST API mostly and sometimes to remember all the endpoints of a service become a challenge.

Also, we write an application on top of a database inorder to sanitize or validate data. We try to save and apply some validation to data to check if they pass some certain criteria and if a user is authorized to perform that activity.

This tool account for those issues with the following advantages:

1. Single point to access data ( inspired by graphql )
2. Permissions and Authorization issues to check if a rule permitted to perform that action.
3. Power to embedded multiple CRUD operation in single request and save round trips.
4. Performing transactions
5. Work with unstructured data like files(e.g images)
6. Query in sync ( return only data that do not exist on local machine ) mode and save bandwidth costs.

Since it uses a single point for entry point we need an expressive rule to explain CRUD operation. Expressive rule will be transferred as JSON from client to server and server will respond with the respective JSON

1.2 HTTP Endpoint

Server must listen to one endpoint by default is /v2 , All Requests will be handled by POST http method. Each request must contain the following header. JSON is used for send and receive messages.

{
  "content-type": "application/json"
}

1.3 Errors

During checking and establishing a connection to start executing rules errors will return as normal HTTP response bodies. When any of the rules result in error that rule value will be null and its error as object will be included in the errors block of the returned data. For example when we create User and Post on the same request and Post fails this is the error.

{
  "createUser": {
    "id": "89695uiu8",
    "createdAt": "2020-02-01 67:89:12 GMT3+"
  },
  "errors": {
    "create.Post": {
      "message": "Post not created because of duplication"
    }
  }
}

2.0 CRUD Rules

Crud operation must be mapped to specific rules for the endpoint to understand and react to specific actions based on a rule.

2.1 Common rules

• Each table/domain/collection must have common attributes which are

  • _id: string -> This must be mapped to id.
  • _created_by: string -> this mapped to createdBy. Which is the id of the user who created that entry to a Domain/Table/Collection.
  • _created_at: Date -> this must be mapped to createdAt.
  • _updated_at: Date -> this must be mapped to updatedAt.

• JSON sent to the server from the client its root must be { } and on top level fields can include the following.

  • Token: string -> which will be used for authorization operation to determine user permission if resource is protected

    {
  "token": "6547fjhfiyr8r"

}

    - **applicationId: string** -> this is a mandatory field for every request

```json
{
  "applicationId": <your-application-id-used-to-start-a-server>
}

- **masterKey: string** -> this is optional field

You can use this to override any rule and perform admin level activities

{
  "masterKey": "654778757bjbo987t876fjhfiyr8r"
}

NOTE: By default a server will not return all attributes when create, update or query but you can override that behavior by set return: [] as its uses will be described in this document. return is a reserved keyword must not be used in as a field or a column in a data you want to save.

When a server successfully serves your request you specify in a rule the response will be available in nameSpace of your rule. Example createUser shall return createUser with your created User.

2.2 Create Domain/Table/Collection

To specify a create/save operation a JSON field must start with a word create then followed by a Domain/Table/Collection name to write data to e.g createUser means add new entry to domain/table/collection called User in database and data to save must be the value of create${domain} field. The word after create must be of any length and cases ideally but following CamelCase will be good for readable code. Examples of create operation will be as follows.

{
  "applicationId": "abs",
  "createUser": {
    "name": "John",
    "age": 30
  }
}

That rule means adding a new entry to the domain/table called User put name=John and age=30. Return id of created data.

{
  "createUser": {
    "id": "6657jg878"
  }
}

To add many data we just send arrays

{
  "applicationId": "6547fjhfiyr8r",
  "createUser": [
    {
      "name": "John",
      "age": 30
    },
    {
      "name": "Doe",
      "age": 12,
      "h": 100
    }
  ]
}

Response should be like.

{
  "createUser": [
    {
      "id": "6657jg878"
    },
    {
      "id": "op65AWjg8"
    }
  ]
}

In the create rule it returns id if you want more than one field to return you must specify those fields with the field return which is an array of extra fields you want to return. For example,

{
  "createUser": {
    "name": "John",
    "age": 30,
    "return": [
      "age",
      "name"
    ]
  }
}

Now the server will respond with the extra field you specify.

{
  "createUser": {
    "id": "6657jg878",
    "name": "John",
    "age": 30
  }
}

If return is empty array like return: [] server will return all data of that document

2.3 Read/Query Domain/Table/Collection

To get or query domain/table/collection a JSON field for that must start with a word query then followed by a Domain/Table/Collection name to query to e.g queryUser means find user(s) to domain/table/collection called User in database and format is query${domain}. The word after query must be of any length and cases ideally but following CamelCase will be good for readable code.

query${domain} rule block has a default field which you can use to shape your query. Those fields are as follows.

• skip: number -> specify data to skip, default is 0.
• size: number -> specify number of data to return, default is 20. If value is negative means you need all the documents to be returned
• count: boolean -> you can count the results of a query to return a number of documents if set to true default value is false.
• orderBy: Array<{field:orderNumber}> -> specify array of fields to order by. orderNumber can be 1 for ascending or -1 for descending. Field is the column we orderBy.
• filter: FilterModel -> this is your query filter you send to server default is {} if not specified
• return: Array -> if not specified default is [] and will return only id. Either you specify or not id return id data match your query found.
• id: string -> if this field present will ignore all other values except return and will return that specific data or null if not found

Examples of query operation will be as follows.

2.3.1 Basic Query

Following example will return all users if exist or [] if no data exist, default return field is id. Since we do not specify any return fields

{
  "queryUser": {}
}

Response from server will be like this:

{
  "queryUser": [
    {
      "id": "op65AWjg8"
    },
    {
      "id": "12sdAWj"
    }
  ]
}

2.3.2 Query By I'd

{
  "queryUser": {
    "id": "op65AWjg8",
    "return": [
      "name",
      "age"
    ]
  }
}

Response from the server will be as follows.

{
  "queryUser": {
    "id": "op65AWjg8",
    "name": "John",
    "age": 30
  }
}

2.3.3 Query By Filter

{
  "queryUser": {
    "filter": {
      "name": "John"
    },
    "return": [
      "age"
    ]
  }
}

Response from the server will be as follows.

{
  "queryUser": [
    {
      "id": "op65AWjg8",
      "age": 30
    }
  ]
}

2.3.4 Count

You can count the result of a query filter and return the total number of matched objects.

{
  "queryUser": {
    "filter": {
      "name": "John"
    },
    "count": true
  }
}

Response from server will be like the following

{
  "queryUser": 1
}

2.3.4 OrderBy

You can order your query result by using the orderBy field inside the queryrule which is an array of maps containing your fields you want to orderBy. The sort map looks like this { : number }, number can be 1 for ascending and -1 for descending. See example below.

{
  "applicationId": "daas",
  "queryProduct": {
    "filter": {},
    "orderBy": [
      {
        "name": 1
      }
    ],
    "return": [
      "name"
    ]
  }
}

Response will be an array of the product array by names in ascending order.

{
  "queryProduct": [
    {
      "name": "apple",
      "id": "y756yh-iuisyu-er56fg"
    }
  ]
}

2.4 Update Domain/Table/Collection

To specify a update/patch operation a JSON field must start with a word update then followed by a Domain/Table/Collection name to write data to e.g updateUser means update entry to domain/table/collection called User in database and data to update must be the value of create${domain} field in JSON you specify. The word after update must be of any length and cases ideally but following CamelCase will be good for readable code.

update${domain} rule block has a default field which you can use to shape your update. Those fields are as follows.

• filter: FilterModel -> update data that match the specified filter object
• size: number -> limit of number of data to search
• skip: number -> number of data to skip when search the match using filter
• id: string -> specify id of the data you want to update if this present filter field will be ignored
• upsert: boolean -> specify if data should be created if not present as a result of a filter, default is false
• update: UpdateModel -> add data you want to update, if not specified default is { }. UpdateModel can contain operator to modify data like insert into a list of array or update a relation
• return: Array -> specify additional fields to return default is []. The operation will return id and updatedAt and with or without those fields you specify in the return field.

2.4.1 Update Many Documents Match Filter Query

This operation will update all the documents that match the given filter and will return all the update documents in array.

{
  "applicationId": "test",
  "updateUser": {
    "filter": {
      "name": "John"
    },
    "update": {
      "$set": {
        "name": "Josh"
      }
    },
    "return": [
      "name"
    ]
  }
}

Response from the server will be like the following.

{
  "updateUser": [
    {
      "id": "98675tgu",
      "updatedAt": "2020-01-97 12:78:00 MT3+",
      "name": "Josh"
    }
  ]
}

2.4.2 Update Single Document By Using Its Id

If you use id to update a document filter and upsert field if present will be ignored and if data with that id is not present will return not found error 404. Examples of update requests will be as follows.

{
  "updateUser": {
    "id": "98675tgu",
    "update": {
      "$set": {
        "name": "Dzeko"
      }
    },
    "return": []
  }
}

Response from server will be as follows

{
  "ResultOfUpdateUser": {
    "id": "98675tgu",
    "updatedAt": "2020-01-97 12:78:00 MT3+",
    "name": "Dzeko",
    "age": 30
  }
}

2.5Delete Domain/Table/Collection

To specify a delete operation a JSON field must start with a word delete then followed by a Domain/Table/Collection name to delete data to e.g deleteUser means delete entry to domain/table/collection called User in database and information of entry to delete must be the value of delete${domain} field in JSON you specify. The word after delete must be of any length and cases ideally but following CamelCase will be good for readable code.

delete${domain} rule block has a default field which you can use to shape your delete operation. Those fields are as follows.

• filter: FilterModel -> delete data that match the specified filter object.
• id: string -> specify id of the data you want to delete, if this present filter field will be ignored.

Delete operation will return only the id of the deleted item ( s ). If delete operation fails will return with error code and message if you delete multiple documents and if a document is not deleted its id field will return null. Example of delete operation is as follows.

2.5.1 Delete Single Document By Id

{
  "deleteUser": {
    "id": "98675tgu"
  }
}

When user is deleted the results will be the id of the delete user

{
  "deleteUser": {
    "id": "98675tgu"
  }
}

2.5.2 Delete MultipleDocument

You can delete multiple data as follows.

{
  "applicationId": "test",
  "deleteUser": {
    "filter": {
      "age": 20
    }
  }
}

The response will be the array of ids of deleted documents or null if the document is not deleted.

{
  "deleteUser": [
    {
      "id": "98675tgu"
    }
  ]
}

3.0 Transactions

Database operations sometimes need to be transactional across multiple Domain/Table/Collection so an easy way to perform transactions is needed. Transaction can be performed by issuing the Transaction Block rule.

You specify the transaction block by transaction keyword and the body can contain many CRUD rule blocks as explained in section 2.

Transaction block has the following field which you put all operation you want to perform

• commit -> This contains all the operations needed for that transaction.

The result of the transaction will be a combination of individual CRUD operations.

Example of transaction operation is as follows.

{
  "applicationId": "6878567gjh",
  "transaction": {
    "commit": {
      "createUser": {
        "name": "Jody"
      },
      "createPayment": {
        "txid": 232353535
      }
    }
  }
}

4.0 Authentication & Permission Policy

Tool must provide a general authentication mechanism for grant access to users and authorization mechanism for resource consumption. Authentication use special domain name to store data which is _User and Authorization it uses _Policy

4.1 Authentication

You use the auth rule block to add new users or get access to the system. auth rule block fields are as follows.

• signUp: SignUpModel -> use this field to register new users to the system.
• signIn: SignInModel -> use this field for already registered users to get a temporary token to access subsequences requests.
• reset: string -> use this to reset a user password using a user email.

4.1.1 Create New User

To create a new user to auth records you must send the following JSON to server

{
  "applicationId": "6878567gjh",
  "auth": {
    "signUp": {
      "email": "eitah12@email.co",
      "username": "eith12",
      "password": "8697tgygyt78tuy",
      "fullName": "8an 9o0"
    }
  }
}

Response from the server will be as follows.

{
  "auth": {
    "signUp": {
      "email": "eitah12@email.co",
      "id": "eitah12968u",
      "username": "eith12",
      "token": "8697tgygyt78tuy.y78967ugkgiyu.099oyu",
      "createdAt": "2020-20-20 &8:09:89 GMT3+",
      "fullName": "8an 9o0"
    }
  }
}

4.1.2 SignIn User

After you get registered next time you need to authenticate yourself, you will send the following request to the server to perform such action.

{
  "auth": {
    "signIn": {
      "username": "eith12",
      "password": "8697tgygyt78tuy"
    }
  }
}

If user successful signIn will return the following response.

{
  "auth": {
    "signIn": {
      "email": "eitah12@email.co",
      "Username": "eith12",
      "id": "78967gkugt",
      "token": "8697tgygyt78tuy.y78967ugkgiyu.099oyu",
      "createdAt": "2020-20-20 &8:09:89 GMT3+",
      "fullName": "8an 9o0"
    }
  }
}

If It fails to log in, the signIn field shall be null and error to be found in the errors block.

4.2 Authorization

Data access policy controlled by rules. You use the policy JSON rule block to add new rules for database access. You need masterKey in the top level block { } to perform this action.

4.2.1 Rule format

The policy block has the following fields, add, remove & list . Rules field is an object which has the following format.

{
  "add": {
    "<resource-operation>": "<condition>"
  },
  "list": {},
  "remove": {
    "ruleId": "<rule-id>"
  }
}

is the operation to act upon a resource, the common operation is

• update.${Domain/Table/Collection}
• create.${Domain/Table/Collection}
• delete.${Domain/Table/Collection}
• query.${Domain/Table/Collection}
• files.save
• files.delete
• files.list
• files.read

is the expression which evaluates to boolean either true or false. In your expression context is an object which will be injected, some of its properties are.

{
  "auth": boolean,
  "uid": string,
  "domain": string
}

• auth -> this field will be true if the user is authenticated otherwise is false.
• uid -> the user id execute the request or undefined.
• domain -> the resource current accessed.

4.2.2 Define Rules

When rules saved will replace any existing rules that match the new rule update. Example of adding rules.

{
  "masterKey": "687tgjhgi78978567gjh",
  "applicationId": "6878567gjh",
  "policy": {
    "add": {
      "query.*": "return true",
      "create.Product": "return false"
    }
  }
}

This symbol * means any Domain/Table/Collection.

Example of defining a javascript function is as follows.

{
  "masterKey": "687tgjhgi78978567gjh",
  "applicationId": "6878567gjh",
  "policy": {
    "add": {
      "query.*": "const auth===context.auth; return auth===true;"
    }
  }
}

The last statement of the condition must return boolean. When return is true means request has access to that resource and if false means request does not have access to that resource.

4.2.3 List Rules

You can list all available rules in your project as follows.

{
  "masterKey": "687tgjhgi78978567gjh",
  "applicationId": "6878567gjh",
  "policy": {
    "list": {}
  }
}

The response from server will be as follows;

{
  "policy": {
    "list": {
      "ruleId": "read.*"
    }
  }
}

4.2.3 Remove Rules

You can remove saved policy by using remove rule block inside policy you will be required to supply the ruleId of the policy you remove. ruleId is equivalent to action you specify when you add a policy rule

{
  "masterKey": "687tgjhgi78978567gjh",
  "applicationId": "6878567gjh",
  "policy": {
    "remove": {
      "ruleId": "query.*"
    }
  }
}

5.0 Aggregation

At some moment we want to perform aggregation of the data instead of just querying with a normal filter. The tool must provide ability to perform aggregation. You use aggregate${Domain/Table/Collection} to perform aggregation to a specified collection. This rule requires a masterKey since aggregation can alter data structure.

{
  "applicationId": "7867tgyujk",
  "masterKey": "785ghjgjfh",
  "aggregateTest": [
    {
      "$match": {
        "name": "qwerty"
      }
    }
  ]
}

If no error the result of aggregation will be found at aggregateTest from the JSON which returned by a server.

6.0 Storage

Server must be able to save a simple file around 5~10 MB and large files with Gb of data and retrieve those files. It has to use amazon s3 compatible storage or GridFS mongoDb storage driver or a custom one as it seems fits.

6.1 Base64 encode files

For simple files we use rule blocks with base64 encoded. To control files you use files rule block, the structure of the rule is as follows.

• save.filename -> this field is any string may contain a file extension for auto content type detect, server will generate a random prefix id and append to the filename you provide so you can use the same file name multiple times and server will return a unique file name for each.
• save.base64 -> the content of file to save base64 encoded or plain text
• save.type -> the content-type mime of the file you save if not supplied mime will be determined from filename extension.
• delete.filename -> filename returned by server not the one you supplied.

{
  "applicationId": string,
  "files": {
    "save": {
      "filename": string,
      "base64": string,
      "type": string
    },
    "delete": {
      "filename": string
    }
  }
}

6.1.1 Save File

To save you send the following JSON to the server.

{
  "applicationId": "d75ujgdkj",
  "files": {
    "save": {
      "filename": "hello.txt",
      "base64": "Helo, world!"
    }
  }
}

If successfully saved, the server will return the path of the file location.

{
  "files": {
    "save": "/storage/778guyg/file/d75ujgdkj/9f8875fb-4064-4d71-584a733-hello.txt"
  }
}

Then you can append the path returned with your server host address to view the file. Or if you know the filename to view the file its path is ${hostname}:${port}/storage/${applicationId}/file/${filename} e.g http://localhost:6000/storage/8767tuy/file/jgyrutiyhjfd-hello.txt

6.1.2 Delete File

To delete a file you send the following JSON to the server.

{
  "applicationId": "d75ujgdkj",
  "files": {
    "delete": {
      "filename": "oioyui-hjgiuguy-jkfjyfh-785ygjkh-hello.txt"
    }
  }
}

If the file is successfully deleted, the server will return the filename which is just deleted.

{
  "files": {
    "delete": "oioyui-hjgiuguy-jkfjyfh-785ygjkh-hello.txt"
  }
}

6.1.3 List Files

To list your files you send the following JSON to the server.

{
  "applicationId": "d75ujgdkj",
  "files": {
    "list": {
      "prefix": ""
    }
  }
}

Server will return array of file objects

{
  "files": {
    "list": [
      {
        "filename": "oioyui-hjgiuguy-jkfjyfh-785ygjkh-hello.txt"
      }
    ]
  }
}

6.1.4 Retrieve a file

When you know the filename you can get file content by using the following format ${hostname}: ${port}/${your-server-endpoint}/storage/${your-application-id}/file/${filename}. For example https://bfast-daas.com/storage/788h/file/76tyuuu-78uui-87ui.mp4

6.2 Upload large files

You can upload large files with efficiency and progress tracking, this will use a direct and dedicated rest api endpoint /storage/${applicationId} with POST http method. This endpoint handle a multipart form data, form your client side you will send your form data with files you want to save and will respond with the urls of the files saved;

The response will be a json of the following format;

{
  "urls": [
    "/storage/889/file/077-hello.txt"
  ]
}

The result will always be an array even if you only upload one file.

7.0 Database Indexes

You can modify the indexes of your database domain/collection/table for performance improvement and other factors as it seems fits. Format of the rule is index${Domain/Collection/Table} e.g indexProduct. This rule requires a masterKey field in your root rule block. Its format is as follows

7.1 Format

The index${domain} block has the following fields, add, remove & list . index field is an object which has the following format.

"add": {"field": "name"},

"list": { },

"remove": { }

7.2 Add Indexes

To add a new index to a domain/table/collection use add which accepts an array of the indexes you want to add, if the index exists will be updated.

{
  "applicationId": "daas",
  "masterKey": "daas-master",
  "indexProduct": {
    "add": [
      {
        "field": "name"
      }
    ]
  }
}

The response from the server will be as follows

{

"indexProduct": { "add": "Indexes added" }

}

7.3 List Indexes

To list all available indexes for a domain use list sub command for index${domain} rule as follows;

{
  "applicationId": "daas",
  "masterKey": "daas-master",
  "indexProduct": {
    "list": {}
  }
}

The response will be the list of the indexes available both user defined and default ( primary key index ).

{
  "indexProduct": {
    "list": [
      {
        "v": 2,
        "key": {
          "name": 1
        },
        "name": "name_1",
        "ns": "051475c8-60f6-4809-a33a-db9db0d4416a.Product"
      }
    ]
  }
}

7.4 Remove Indexes

You can delete/remove a user defined index only by using the remove rule in index${domain} rule block.

{
  "applicationId": "daas",
  "masterKey": "daas-master",
  "indexProduct": {
    "remove": {}
  }
}

The response from the server will be as follows.

{
  "indexProduct": {
    "remove": true
  }
}

8.0 PlayGround

This is an interface for you to play with your rules when developing your application. You open the UI by running the following command.

~$ bfast database playground

You must install " bfast-tools" a cli tool from npm like this "~$ npm install -g bfast-tools"

Other Reference

1. bfast-tools
2. bfast-function
3. bfast website
4. playground website