express-monger v1.1.0

express-monger

Yet, another package to streamline the creation of a REST API based on MongoDB and Express.js

Contents

• Getting Started
  • Prerequisites
  • Installing
• Usage
• CRUD Operations
  • All resources
  • Databases
  • Collections
  • Documents
• Configuration
• ClientsPool
• Authentication
  • Credentials
  • Auth request
• Features
  • HATEOAS
  • Pagination
  • Filtering
  • Sorting
  • Projections
  • Embedding
  • Distinct
  • Aggregate
  • Pretty printing
  • Force
  • Batch Insert
  • Batch Delete
  • Batch Update
• Versioning
• Authors
• Acknowledgments

Getting Started

express-monger is package to ease the creation of REST API for MongoDB using Express.js. It is built aiming to be as flexible, customizable and low-dependency as possible, providing nonetheless powerful functionalities like pagination, filtering or using HAL principle for the responses.

Prerequisites

The package essentially provides a set of middleware methods to add CRUD operations to an Express.js app, based on MongoDB. Therefore, a minimum set up is required:

• MongoDB, either locally or cloud based. Version 3.1 or above;
• Node
• Express, installed via npm
• Body-parser, installed via npm

Installing

npm install express-monger

Usage

The package is built to be quite flexible and ready to use. A quick start setup:

/** index.js */

const express = require('express');
const bodyParser = require('body-parser');
const Monger = require('express-monger');

let monger = new Monger("mongodb://localhost:27017", {
    entryPoint: "api",
    showAll: true
});

let app = express();
app.use(bodyParser.json());

// Hardcoded credentials to be used.
// This can be replaced with Passport.js or any
// other authentication package.
app.use((req, res, next) => {
    req.user = {
        username: "test-user",
        password: "testing123456"
    };

    next();
});

monger.initialize(app);

app.listen(2000, () =>{
    console.log("express-monger up and running on port 2000!");
});

Now, run index.js file.

node index.js

The above example creates a server on locahost:2000 and following the appropriate URL structure, databases, collections and documents are read-only accessible (GET method) out of the box, as long as authentication is not required or the user has access to the resources.

CRUD Operations

express-monger generates a set of CRUD operations ready to use on Mongo's databases, collections and documents, as long as they have been specified on the config options. If not, all end points can be accessed only by using GET request.

All resources

If the option showAll is enabled on the configuration and the authenticated user has admin rights, all databases and their collections can be accessed through a GET request via:

• /{entryPoint}/{allEndPoint} if both parameters are set up on the configuration.
• /{entryPoint} or /{allEndPoint} if entryPoint or allEndPoint are set up respectively.
• / if none of the above.

Databases

The URL for perform actions to databases follows the structure /:database or /{entryPoint}/:database if entryPoint has been defined.

Collections

The URL for perform actions to collections follows the structure /:database/:collection or /{entryPoint}/:database/:collection if entryPoint has been defined.

Other collection endpoints:

• /:database/:collection/distinct. See Distinct.
• /:database/:collection/aggregate. See Aggregate.

Documents

The URL to perform actions to documents follows the structure /:database/:collection/:_id or /{entryPoint}/:database/:collection/:_id if entryPoint has been defined.

Configuration

When instantiating a new Monger object, the only required value is the MongoDB URI where to connect. The rest are optional configurations to customize how the API behaves.

[...]

let config = {
    conTimeout: 10000,
    credentials: {
        key: 'user',
        usernameField: 'email',
        passwordField: 'password'
    },
    databaseMethods: ['get', 'post'],
    collectionMethods: ['get', 'post'],
    documentsMethods: ['get', 'put', 'patch', 'delete'],
    maxResult: 50,
    maxLimit: 500,
    entryPoint: 'api',
    showAll: true,
    allEndPoint: 'my-mongo'
};

let monger = new Monger("mongodb://localhost:27017", config);

[...]

Some WIP features under consideration to be implemented:

ClientsPool

Since MongoDB Node Driver 3.6, the option of authenticating multiple credentials under the same connection pool was removed, so the authentication needs to happen when instantiating the MongoClient by adding the username and password to the URI (see authentication example). This means that for every authenticated request a new connection gets created, which can result on performance penalties and even collapsing Mongo if the number of concurrent connections increases to warning levels.

A ClientsPool is merely a way of keeping connections opened, getting closed and discarded after a defined time if they haven't been used again during that period.

• If an authenticated request does not have an active/open connection, it will be created and saved with an expire date equal to the creation date plus the predefined time.
• If an authenticated request does have an active/open connection, this one is used after updating its expire date.

When a ClientsPool is created, a background loop process is created and run at intervals to check if any of the active/open connections has expired already. If so, it gets closed and removed from the pool.

The ClientsPool instance is exposed under monger.clientsPool after the monger objects has been instantiated.

Authentication

express-monger does not provide authentication methods by itself, but relies on user's credentials in order to connect to the MongoDB instance if such is required.

Users accessibility and role are defined by Mongo, so access levels and roles should be defined per user on the Mongo instance itslef.

Credentials

The package will look into the req object for an object that should contain the username and password required to authenticate to Mongo. By default, the package will look to the user object:

// Default structure of the object to look for the credentials
req.user: {
    username: "the_user",
    password: "the_password"
}

This can be customized by providing the structure of the object expected on the configuration options.

// Setting this at the configuration options
config.credentials = {
    key: "userCredentials",
    usernameField: "email",
    passwordField: "secret_word"
}

[...]

// The module will look for this object when getting the credentials
req.userCredentials: {
    email: "the_user",
    secret_word: "the_password"
}

Auth request

For small, one-person applications, it can be enough to set the credentials on a middleware method before initializing the monger instance.

[...]

app.use((req, res, next) => {
    req.user = {
        username: "my-username",
        password: "my-password"
    };

    next();
});

monger.initialize(app);

[...]

Another option for mid-to-large apps is to use a third party authentication package. Passport, for example, is a quite solid package with multiple methods providing a high level authentication process.

This example works by trying to create a new connection to MongoDB with the provided credentials. The ClientsPool is used in order to save the connection if it succeeds, so it can be used later on the request instead of opening and closing it.

const passport = require('passport');
const Monger = require('express-monger');

[...]

app.use(passport.initialize());

// Monger instance using a URI set on the global scope
let monger = new Monger(process.env.MONGODB_URI);

// From Passport.js documentation. Logic needs to be implemented accordingly
passport.use('basic', new BasicStrategy(
  async function(monger.clientsPool, username, password, done) {
    try{
        var options = {
            useNewUrlParser: true
        };
        var client = await clientsPool.connect(process.env.MONGODB_URI, username, password, options);

        if (client.isConnected()){
            return done(null, {'username': username, 'password': password});
        }else{
            throw new Error();
        }
    } catch(e){
        return done(null, false, { message: 'Authentication failed' });
    }
  }
));

// Setting Passport middleware before any rout requiring authentication
app.use(passport.authenticate('basic',
    {
        session: false
    }
);

// monger.initialize(app) generates a bunch of routes requiring authentication,
// so it needs to be called after Passport's middleware.
monger.initialize(app);

[...]

Features

The package provides some nice features detailed below.

HATEOAS

Hypermedia as the Engine of Application State (HATEOAS) is enabled by default and closely follows the HAL standard. When a GET request is made, the JSON response will have a "_links" property containing details to other resources, with a href link relative to the host and a title for each one.

It also has an "_items" property which contains an array of the returned elements, and "_meta" which contains information of the pagination (if enabled, see Pagination) and total number of items on the resource.

When HATEOAS is enabled, the response's Content-type will be application/hal+json, and application/json when disabled.

curl --user username:password -i "http://localhost:2000/company/assets?pretty"
HTTP/1.1 200 OK
Content-Type: application/hal+json; charset=utf-8

{
    "_links": {
        "self": {
            "href": "/company/assets",
            "title": "a-collection"
        },
        "parent": {
            "href": "/company",
            "title": "company"
        },
        "find": {
            "href": "/company/assets/{id}",
            "templated": true
        },
        "next": {
            "href": "/company/assets?page=2",
            "title": "next page"
        },
        "last": {
            "href": "/company/assets?page=48",
            "title": "last page"
        }
    },
    "_items": [...],
    "_meta": {
        "max_results": 100,
        "total": 4725,
        "page": 1
    }
}

HATEOAS can also be disabled on a particular request by adding a ?hateoas=false parameter to the resquested URL, or enabled if globally disabled with ?hateoas=true.

# Will disable hateoas for this request, if it is globally enabled
curl "http://localhost:2000?hateoas=false"

# Will enable hateoas for this request, is it is globally disbaled
curl "http://localhost:2000?hateoas=true"

NOTE

Disabling HATEOAS can be sensible if the end user will not require it, as it will save bandwidth and improve performance. But if pagination is enabled, user will not have any information on how to navigate the resource and result on a poor and failing interaction with the database.

Pagination

Collections can have thousands of documents than when retrieved can result on running out of memory on the client side and crash. Server-side pagination comes to the rescue by capping the number of results returned by a GET call.

The default value is 100 items, although this can be changed by the property maxRestuls on the Configuration options. This can be overwritten by adding a ?max_results= parameter to the URL. As a safety method, the maximum results a user can request is also capped by the maxLimit property, being 1000 the default.

# This returns a maximum of 100 items, or as defined on the configuration options
curl "http://localhost:2000"

# This returns a maximum of 500 items
curl "http://localhost:2000?max_results=500"

# This returns a maximum of 1000 items regardless of the query parameter, or as defined on the configuration options
curl "http://localhost:2000?max_results=3000"

Navigating through pages is as easy as adding a ?page= query parameter.

# Will return the second page, if any, of a collection with a limit
# of 100 (or set up value) items per page
curl "http://localhost:2000?page=2"

# Will return the third page, if any, of a collection with a limit
# of 500 items per page
curl "http://localhost:2000?max_results=500&page=3"

Pagination can also be turned off on a particular request by adding a ?pagination=false parameter to the resquested URL, or turn it on if globally disabled with ?pagination=true.

# Will disable pagination for this request, if it is globally enabled
curl "http://localhost:2000?pagination=false"

# Will enable pagination for this request, is it is globally disabled
curl "http://localhost:2000?pagination=true"

NOTE

Disabling pagination is not recommended as it can cause an unresponsive state and crash the client-side browser. Only disable if you know the number of documents can be handled by the browser.

Filtering

When querying documents from a collection, the response can be filtered by simply using a ?where= parameter on the request. This value only accepts Mongo's native querying formats on JSON format, including regular expressions.

For example, this GET request:

"http://localhost:2000/company/people?where={'lastname':'Doe','firstname':'/^J/i'}"

will return all documents on the people collection from the company database, where the lastname is equal to Doe and firstname starts with J or j.

// response sample
{
    "_links": {
        "self": {
            "href": "/company/people?where={\'lastname\':\'Doe\' , \'firstname\':\'/^J/i\'}",
            "title": "people"
        },
        "parent": {
            "href": "/company",
            "title": "company"
        },
        "find": {
            "href": "/company/people/{id}",
            "templated": true
        }
    },
    "_items": [
        {
            "_id": "5b6182a5be180b8d888a2dbe",
            "firstname": "Jane",
            "lastname": "Doe",
            "age": 32
        },
        {
            "_id": "5b6182a5be180b8d888a2dc3",
            "firstname": "john",
            "lastname": "Doe",
            "age": 29
        }
    ],
    "_meta":{
        "max_results": 100,
        "total": 2,
        "page": 1
    }
}

NOTE

The above request will work fine on a browser, but will need to be URI encoded using curl on the command line:

curl "http://localhost:2000/company/people?where=%7B'lastname':'Doe','firstname':'/%5EJ/i'%7D"

Sorting

Responses can be sorted using a ?sort= query parameter, which accepts two syntaxes:

• Mongo native syntax: [("lastname", 1),("age", -1)]
• Custom syntax: lastname,-age

Both will return a sorted response by first ascending lastname and then descending age.

# On browser
"http://localhost:2000/company/people?sort=[['lastname',1],['age',-1]]"
"http://localhost:2000/company/people?sort=lastname,-age"

# CURL
curl "http://localhost:2000/company/people?sort=%5B('lastname',1),('age',-1)%5D"
curl "http://localhost:2000/company/people?sort=lastname,-age"

Projections

Projections controls the fields returned from querying collections and documents.

• Using the projection query param, a native Mongo syntax can be used, like ?projection={"lastname":1, "age":1}.
• Using the fields query param, a simpler syntax can be used which depends only on the fields to be returned or not, for example ?fields=lastname,age

# Examples

# This will only return field lastname from the documents if field exists.
"http://localhost:2000/company/people?fields=lastname"

# This will return all fields except age and _id
"http://localhost:2000/company/people?sort=-age,-_id"

Embedding

A document embedding mechanism is implemented, using MongoBD's DBref feature. Please refer to DBRef to learn how to refer to a document from another document.

When making a GET request to a document or collection, the embedded query parameter can be used to retrieve reference documents as if they were embedded in the main document, if the value of the given field names is a DBRef, the document it refers to is replaced at runtime into the document(s).

Without embedded:

curl --user username:password -i "http://localhost:2000/company/employee?pretty"
HTTP/1.1 200 OK
Content-Type: application/hal+json; charset=utf-8

{
    "_links": {...},
    "_items": [
        {
            "name": "John",
            "office":{
                "$ref": "offices",
                "$id": "5be18c9def3f9050ac59612f"
            },
            [...]
        },
        {...}
    ],
    "_meta": {...}
}

With embedded:

Multiple keys can also be used as the value of embedded parameter if they are separated by comma "," character.

Reference documents can also have documents referenced inside them. In that case dot notation “.” should be used.

curl --user username:password -i "http://localhost:2000/company/employee?embedded=office&pretty"
HTTP/1.1 200 OK
Content-Type: application/hal+json; charset=utf-8

{
    "_links": {...},
    "_items": [
        {
            "name": "John",
            "office":{
                "_id": "5be18c9def3f9050ac59612f",
                "location": "London",
                "postcode": "WC1X 8AL",
                [...]
            },
            [...]
        },
        {...}
    ],
    "_meta": {...}
}

This method is very convenient when making GET requests, but please note this is not be the most efficient in terms of size of the data transferred between the API and the client. If you have many to one relationships in your data model where a few documents are referenced by a lot of other documents, referenced objects are transferred as many times as their referrer documents.

Distinct

The distinct endpoint on collections provides an easy and fast action to retrieve all unique values associated with given keys on the collection.

curl --user username:password -i "http://localhost:3000/test/people/distinct?fields=firstname,lastname&pretty"
HTTP/1.1 200 Ok

{
    "firstname": [
        "Jane",
        "Michael",
        "Grigori",
        "Gavrilo",
        "Franz"
    ],
    "lastname": [
        "Howard",
        "Kane",
        "Peshkov",
        "Princip",
        "Ferdinand",
        "Doe"
    ]
}

Aggregate

The aggregate endpoint on collections provides an interface to execute MongoDB Aggregation Pipelines.

It requires a pipeline query parameter of type array.

// Example of simple aggregation on javascript.
[
    "$group":{
        _id: "null",
            averageAge: {$avg: "$age"}
        }
]

curl --user username:password -i "http://localhost:3000/test/people/aggregate?pipeline=[\"$group\":{\"_id\":\"null\",\"averageAge\":{\"$avg\":\"$age\"}}]&pretty"
HTTP/1.1 200 Ok

{
    "_id":"null"
    "averageAge":35.625
}

Pretty printing

The response can be pretty printed (basically human readable) simply by adding pretty as a query parameter.

curl --user username:password -i "http://localhost:3000/test/people?pretty"
HTTP/1.1 200 OK

{
    "_links": {
        "self": {
            "href": "/test/people",
            "title": "people"
        },
        "parent": {
            "href": "/test",
            "title": "test"
        },
        "find": {
            "href": "/test/people/{id}",
            "templated": true
        }
    },
    "_items": [
        {
            "_links": {
                "self": {
                    "href": "/test/people/5bb4ca7bd0381f0b649fa3f3"
                }
            },
            "_id": "5bb4ca7bd0381f0b649fa3f3",
            "firstname": "John",
            "lastname": "Doe",
            "age": "29"
        },
        {
            "_links": {
                "self": {
                    "href": "/test/people/5bb4ca87d0381f0b649fa3f4"
                }
            },
            "_id": "5bb4ca87d0381f0b649fa3f4",
            "firstname": "Jane",
            "lastname": "Doe",
            "age": "32"
        },
        {
            "_links": {
                "self": {
                    "href": "/test/people/5bb4ca9bd0381f0b649fa3f5"
                }
            },
            "_id": "5bb4ca9bd0381f0b649fa3f5",
            "firstname": "Michael",
            "lastname": "Kane",
            "age": "45"
        }
    ],
    "_meta": {
        "max_results": 100,
        "total": 3,
        "page": 1
    }
}

Force

Some actions like DELETE or PUT can be dangerous to perform, so upon request they will return an error if the requiresForce property is enable on the configuration.

curl --user username:password -X "DELETE" -i "http://localhost:3000/test/people/5bb4ca7bd0381f0b649fa3f3?pretty"
HTTP/1.1 400 Bad Request

{
    "_status":400,
    "_message":"To pertmantly delete the document, include parameter `force` on the URL request"
}

Including the force query parameter to the request will perform the action, considering the user is allowed to.

curl --user username:password -X "DELETE" -i "http://localhost:3000/test/people/5bb4ca7bd0381f0b649fa3f3?pretty&force"
HTTP/1.1 204 No Content

Batch Insert

When requesting a POST on a collection a body needs to be sent, either a single element or an array of them. It will failed if at least one if an empty object.

Documents

Single document example

curl --user username:password -X "POST" -d "{\"firstname\":\"Grigori\",\"lastname\":\"Peshkov\"}" -H "Content-Type: application/json" -i "http://localhost:2000/company/people?pretty"
HTTP/1.1 201 Created
Content-Type: application/hal+json; charset=utf-8

{
    "_status": 201,
    "_message": "1 document created.",
    "_items": [
        {
            "_id": "5bb4e9726ff63b2540c2f6ed",
            "href": "/company/people/5bb4e9726ff63b2540c2f6ed"
        }
    ]
}

Multiple documents example

curl --user username:password -X "POST" -d "[{\"firstname\":\"Gavrilo\",\"lastname\":\"Princip\"}, {\"firstname\":\"Franz\",\"lastname\":\"Ferdinand\"}]" -H "Content-Type: application/json" -i "http://localhost:2000/company/people?pretty"
HTTP/1.1 201 Created
Content-Type: application/hal+json; charset=utf-8

{
    "_status": 201,
    "_message": "2 documents created.",
    "_items": [
        {
            "_id": "5bb4eacd6ff63b2540c2f6ee",
            "href": "/company/people/5bb4eacd6ff63b2540c2f6ee"
        },
        {
            "_id": "5bb4eacd6ff63b2540c2f6ef",
            "href": "/company/people/5bb4eacd6ff63b2540c2f6ef"
        }
    ]
}

Collections

Single or multiple collections can be created by making a POST request to a database end point.

The body sent needs to be one or an array of objects having, as a minimum, a name property. An optional options property can also be sent, being these exactly as defined on the createCollection method.

curl --user username:password -X "POST"  -d "[{\"name\":\"foreign-accounts\"},{\"name\":\"foreign-assets\"}]" -H "Content-Type: application/json" -i "http://localhost:3000/company?pretty"
HTTP/1.1 201 Created
Content-Type: application/hal+json; charset=utf-8

{
    "_status": 201,
    "_message": "2 collections created.",
    "_items": [
        "foreign-accounts",
        "foreign-assets"
    ]
}

Batch Delete

Deleting a single document or a single collection is as simple as making a DELETE request to /:database/:collection/:_id and /:database/:collection respectively. But if we need to delete multiple documents, let's say filtered by some values, it would be a pain to first GET all the filtered documents and then make multiple DELETE request for each _id.

Documents

Taking advantage of filtering, in order to delete multiple documents, we could append the ?where= query parameter to the collection end point, and it will swap from dropping the collection to removing the matching documents.

If configuration option requiresForce is enabled, it will return a 400 Bad Request, expecting the force parameter to be passed.

## This example tries to delete all documents which versions is greater than 40 and less than 50.
curl -g --user username:password  -X "DELETE" -i "http://localhost:3000/company/assets?where={\"version\":{\"$gte\":40,\"$lte\":50}}&pretty"
HTTP/1.1 400 Bad Request
X-Powered-By: Express
Content-Type: application/hal+json; charset=utf-8

{
    "_status":400,
    "_message":"To pertmantly delete 1133 documents out of 7722, please include parameter `force` on the URL request"
}

Adding force will permantly delete the filtered documents.

curl -g --user username:password  -X "DELETE" -i "http://localhost:3000/company/assets?where={\"version\":{\"$gte\":40,\"$lte\":50}}&force"
HTTP/1.1 204 No Content

Collections

Deleting multiple collections follows the same principle of using filtering on a database end point, but only accepting the name field to be queried.

If configuration option requiresForce is enabled, it will return a 400 Bad Request, expecting the force parameter to be passed.

## This example tries to delete all collections which name starts with `Foreign` or `foreign`.
curl -g --user alvaro-ortega-pickmans:snhXJ5eKeAsCXu7J -X "DELETE" -i "http://localhost:3000/company?where={\"name\":\"^/foreign/i\"}"
HTTP/1.1 400 Bad Request
Content-Type: application/hal+json; charset=utf-8

{
    "_status": 400,
    "_message": "To pertmantly delete 2 collections, please include parameter `force` on the URL request"
}

Adding force will permantly delete the filtered collections.

curl -g --user username:password -X "DELETE" -i "http://localhost:3000/company?where={\"name\":\"^/foreign/i\"}&force"
HTTP/1.1 204 No Content

NOTE

Disabling requiresForce will not warn the user about the deletion and will remove all documents/collection matching. If an empty object is passed with the where parameter, every element will match and therefore be deleted. Use with extremely care.

Batch Update

Following similar approach as on Batch Insert and Batch Delete, express-mogner makes benefits from the filtering mechanism to allow multiple documents updates.

This feature only works on collection endpoints, being the body of the request a JSON string following Mongo's update operators structure.

curl --user username:password -X "PATCH" -d "{\"$set\":{\"status\":\"Married\"}, \"$currentDate\":{\"modified\":true} }" -H "Content-Type: application/json" -i "http://localhost:2000/company/people?where={'lastname':'Doe'}&pretty"
HTTP/1.1 201 Created
Content-Type: application/hal+json; charset=utf-8

{
    "_status": 201,
    "_message": "2 documents updated."
}

Versioning

SemVer is used for versioning.

Authors

• Brydenwood CreativeTech @ Bryden Wood

Acknowledgments

Most of the pacakge's structure and functionality is inspired and based on Python's Eve framework.