Darc-mongoreadyserver NPM

Introduction

The purpose of this repo is to make it "easy" to create useful, functional backends in node.js for full-stack application building. The underlying database is assumed to be MongoDB.

Features

• Runnable: Easy specification of how your server should run with environment variables, configuration files, and/or command line arguments. These are handled by a "sibling" package, darc-mongoreadyserverconfig whose function is only to handle accepting, parsing, and validating arguments.

• Connected to MongoDB: The package manages connection to a (specified) database and set of collections for you, so all you have to know is the MongoDB node.js API calls for collections. That is, routines like find, updateOne, deleteMany, etc.

• Configurable: Automatic loading of additional, app-specific options from MongoDB (so that operation isn't dependent on machine-specific configuration settings).

• Securable: http or https request handling, depending on whether you specify an SSL cert/key pair or not.

• Securable: Request authorization and rights- or role- based access control (RBAC). By default authorization is simply informative, but can be made restrictive (requests get denied if not authorized) and granular (with RBAC). RBAC handling is up to you, but can be plugged into the middleware installed in this server.

• Useful Logging: Every request and response is logged, as is every error, with (hopefully) useful data. All requesters are also logged, as identified with Authorization and Identity headers in requests. Each log item is written to a single log file, but prepended with useful tags to enable easy sorting and filtering, and every request is coded with a unique key so that log lines for separate events tied to one request are "joinable".

• Customizable: you specify a single folder in which to look for additional "modules" to load, containing what routes to define, what tests to compile, plus more. This package loads all of this for you if you follow the conventions described below.

• Reloadable: There is a built-in route that can call for a full server reboot, including renewing the connection to MongoDB. This way, you can update options and adopt them with a reboot without logging into the machine running a server. • Autonomously Auditable: We've built in a "check loop" feature that allows you to specify anything you might need to check on a schedule, say daily at 1 AM, to ensure your server is functioning correctly and/or report back on its availability and performance. You define what, and when, to check.

• Testable: Define tests to run right alongside the routes they should test. darc-mongoreadyserver will combine all these specs for you into a script you can run, as well as a route to get these tests. Writing tests right next to their handlers is a nice design pattern.

• Lightweight: darc-mongoreadyserver doesn't itself rely on a billion packages. It uses the external packages express, cors, axios, log4js, mongodb, and mongoose, and the internal (but publicly available) packages daat-coordinator and darc-mongoreadyserverconfig. These each depend on other packages of course, but the overhead is small. A clean install is about 16MB.

• Deployable: darc-mongoreadyserver can create template systemd service unit files for you, to make a production deployment easy.

Usage

Our intended usage is, hopefully simple.

Trivial Example

Here's probably the most trivial example. Put the following in index.js (or whatever) in your node.js project:

require( 'darc-mongoreadyserver' )( )
	.catch( err => { console.log( "FAILED TO LAUNCH SERVER: " , err ); } );

Note that the package exports a function returning a Promise. (Promises are great, if you don't know about them you should learn.) For this to work you also, of course, have to run

$ npm install --save darc-mongoreadyserver

Running this alone produces

$ node index.js
[2020-02-07T16:43:49.788] [INFO] info - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 
[2020-02-07T16:43:49.790] [INFO] info - * myproject (v 1.0.0) starting...
[2020-02-07T16:43:49.790] [INFO] info - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 
FAILED TO LAUNCH SERVER:  Configuration passed to mongodb.reloadMongoDB must have a mongodb field with connection information.

Note a couple things: First, the "fancy" logging is automatic, and darc-mongoreadyserver reads your package file for some info about your project written to the logs. Second, you have to have and specify a database for this to run, and the package fails without it. Sorry, we don't make and host db's for you.

Presuming you have one, and the associated connection string, you can simply run

GSB-C02TF2BAGTFL:myproject morrowwr$ node src/index.js --mdb-uri "mongodb+srv://${USERNAME}:${PASSWORD}@mymongodb-8hgas.mongodb.net/myproject-database"
[2020-02-07T18:17:13.249] [INFO] info - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 
[2020-02-07T18:17:13.252] [INFO] info - * myproject (v 1.0.0) starting...
[2020-02-07T18:17:13.252] [INFO] info - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 
[2020-02-07T18:17:14.257] [INFO] info - Waiting for MongoDB connection...
[2020-02-07T18:17:19.262] [INFO] info - Connected to MongoDB.
[2020-02-07T18:17:20.238] [INFO] info -   Loaded collection "options" (from options)
[2020-02-07T18:17:20.238] [INFO] info -   Loaded collection "errors" (from errors)
[2020-02-07T18:17:20.238] [INFO] info - Using most recent options stored in MongoDB
[2020-02-07T18:17:20.545] [INFO] info - check loop interval: 86400000
[2020-02-07T18:17:20.548] [INFO] info -   loading module custom
[2020-02-07T18:17:20.553] [INFO] info - Offset to regular check time: 20559447 ms (342.7 min, 5.7 hr)
[2020-02-07T18:17:20.553] [INFO] info - Listening on port 5000
[2020-02-07T18:17:20.554] [INFO] info - Got context back in importing application.

and get something like the above. Obviously you'll need to specify your own connection string.

You can call the basic routes provided, like

$ curl localhost:5000 
myproject (1.0.0): a sample project for testing darc-mongoreadyserver.

where these fields are drawn from the package.json file in your project. Or

$ curl localhost:5000/options

which will list out all the configuration passed and options stored in your MongoDB instance, or

$ curl -w "\nStatus Code: %{http_code}\n" localhost:5000/fake/error
Error: fake error called.
Status Code: 500

Adding Routes with Custom Modules

How would we add a custom route? First, make a directory modules at the same level as index.js. Second, change the call to the package in index.js to be:

require( 'darc-mongoreadyserver' )( __dirname + '/modules' )
	.catch( err => { console.log( "FAILED TO LAUNCH SERVER: " , err ); } );

See the argument passed? This is telling the package to look in this modules folder for things it should try to load. It will try to load any js file in that directory. So let's make one! Put

let routes = {};
routes.simple = {
	methods : 'get' , 
	route : '/custom' , 
	callback : ( req , res ) => {
		res.send( "YES! This is a custom route.\n" );
	}
};
module.exports = ( ctx ) => {
	return {
		routes : routes , 
	}
};

in a file modules/custom.js. The package should load this automatically, print a corresponding message, and running

$ curl localhost:5000/custom
YES! This is a custom route.

And that's all we had to do!

To be a bit more belaboured about what we've done, we've created (and ultimately exported) a routes object whose keyed values are themselves objects specifying the http method to use for a route defined by the route field, for which the handler is given by the callback field. The corresponding plain express action would be

app.get( '/custom' , ( req , res ) => { res.send( "YES! This is a custom route." ); } );

and this is what the package does for you.

Note, though, that your custom module has to actually export a function that takes in the context and returns an object with the "actual" exports.

Reviewing Server Logs

We automatically get logging for requests like these. Specifically, you should see something like the following in the terminal running the server:

[2020-02-07T17:59:10.140] [INFO] info - REQLOG,40daf41b01c11164bd958d36,GET,/custom,1581127150.140
[2020-02-07T17:59:10.141] [INFO] info - WHOLOG,40daf41b01c11164bd958d36,unknown,unknown
[2020-02-07T17:59:10.141] [INFO] info - RESLOG,40daf41b01c11164bd958d36,GET,/custom,200,1581127150.140,1581127150.141,0.001

Lines with REQLOG log "requests", lines with WHOLOG log identity and authorization for requests (if they exist), and lines with RESLOG log responses. Note each has a key, 40daf41b01c11164bd958d36, enabling comparison even when log lines for different requests are interleaved. REQLOG lines and RESLOG define the call made to the server, REQLOG lines also include the epoch time a request was recieved, while RESLOG lines include the response status code, the receipt time, the response time, and the associated duration of the request-response cycle.

Now try this:

$ curl localhost:5000/custom -H 'Authorization: none' -H 'Identity: hidden'
YES! This is a custom route.

The associated log lines are:

[2020-02-07T18:04:39.601] [INFO] info - REQLOG,cc6f5d8434c85671130ff625,GET,/custom,1581127479.601
[2020-02-07T18:04:39.601] [INFO] info - WHOLOG,cc6f5d8434c85671130ff625,hidden,none
[2020-02-07T18:04:39.601] [INFO] info - RESLOG,cc6f5d8434c85671130ff625,GET,/custom,200,1581127479.601,1581127479.601,0.000

Note here that the WHOLOG line writes out the values of the Authorization and Identity headers passed. This way, we can filter requests to particular users with Identity, and even specific "sessions" should our Authorization tokens speak to that (as they do with rsslogin). If you write a front end that makes request to the backend, you should make sure these headers are set for each request.

The Context

Did you notice we defined the module.exports in our modules/custom.js as a function with an argument? Like this:

module.exports = ( ctx ) => { ... };

This "ctx" is a "context" object passed from the package containing the following fields:

dir: the directory you're project is really running from.
pkg: the content of your package.json, as an object.
cfg: the full configuration for the server, including MongoDB-sourced options.
log: the logger object, for making your own writes to the logs.
utl: utility functions included in darc-monngoreadyserver.
col: the simplified "collections" object referencing MongoDB collections you might use.
mdb: a MongoDB object defined by darc-monngoreadyserver.
srv: the server object and, more importantly, the routine srv.defineRBACModule that you can use to install RBAC.
mod: an object containing the ultimately-exported objects from all the custom modules you say to include.

cfg, log, col, and utl are probably the most useful parts of the context, along with mod if your custom modules are interrelated. You may need cfg to customize how routes load or act, according to your needs. Use log to write to logs in a consistent fashion. col intends to provide an easy way to access different collections you might need to interact with in your project. Finally utl contains some routines that may be useful to your project.

Technically this isn't necessary. Your custom modules can export a plain object, but such objects cannot have any dependencies on the context. They basically have to be constants or otherwise completely independent of the infrastructure in darc-mongoreadyserver (including other custom modules).

Adding RBAC

This is a simple syntax for importing darc-mongoreadyserver when we want to use some sort of RBAC:

require( 'darc-mongoreadyserver' )(  ) // using default module location
	.then( ctx => { ctx.srv.defineRBACModule( ctx.mod.rbac ); } )
	.catch( err => { console.log( "FAILURE:: " , err ); } )
	.finally( process.exit )

Note all we have to do is tell the "server" part of darc-mongoreadyserver what module we want to use to handle RBAC.

Specifically, we include a custom module, defined in modules/rbac.js (or wherever your custom modules are), that we declare (after everything has loaded) as the module to use to implement RBAC. All this module has to do is return from it's export function (yes function; see above) at least the function allowRequest. This function should take in the request and return a boolean signifying whether the request should be allowed (true) or not (false). darc-mongoreadyserver handles the actual NotAuthorized response from here. This method allowRequest will be called for every request, as it is installed as middleware, so whatever it does it should be efficient. The rest is up to you!

Note also that the RBAC middleware follows the login/authorization middleware. Any request processed by allowRequest will be "authorized" at that level already, if you are enforcing authorization.

Here's a tangible, real example of such an rbac module from one of our real production applications:

const _RDS = require( 'route-data-store' );
const rightsMap = { admin : 3 , approver : 2 , authed : 1 , anyone : 0 };
var _config , _log , _roles , routes = {};

module.exports = ( ctx ) => {

	// get the objects we need from the context
	_config = ctx.cfg; _log = ctx.log; _roles = ctx.mod.roles;

	// scan ALL modules in the context for routes, and compile a list of the rights required
	// for them. THIS MEANS RBAC HAS TO RUN ** LAST ** IN THE LOADING STACK. 
	Object.keys( ctx.mod ).forEach( m => {
		if( "routes" in ctx.mod[m] ) {
			Object.keys( ctx.mod[m].routes ).forEach( r => {
				let methods = ctx.mod[m].routes[r].methods;
				let route   = ctx.mod[m].routes[r].route;
				let rights  = ctx.mod[m].routes[r].rights;
				if( ! rights ) { rights = "anyone"; } // default to "anyone" call call this route spec
				if( Array.isArray( methods ) ) {
					methods.forEach( method => { 
						method = method.toUpperCase();
						if( ! ( method in routes ) ) { routes[method] = new _RDS(); }
						routes[method].addData( route , rightsMap[rights] );
					} );
				} else {
					method = methods.toUpperCase();
					if( ! ( method in routes ) ) { routes[method] = new _RDS(); }
					routes[method].addData( route , rightsMap[rights] );
				}
			} );
		}
	} );
	
	return { 
		allowRequest : ( req ) => {
			// 1. define the request caller's rights, embedding that data in the request object
			if( _roles.isadmin( req.identity ) ) { req.rights = "admin"; }
			else if( _roles.isapprover( req.identity ) ) { req.rights = "approver"; }
			else { req.rights = "authed"; } // if we are here, we are authorized
			// 2. get required rights (efficiently) from tree data structure created above
			let requiredRights = routes[req.method].getData( req.originalUrl );
			// 3. get request rights from the request object
			let requestRights  = ( typeof req.rights === "undefined" ? 0 : rightsMap[req.rights] );
			// 4. return true if there are sufficient rights? 
			return ( requestRights >= requiredRights );
		} 
	}

}

Here we presume there is a module roles that is loaded before rbac, a module with functions "isadmin" and "isapprover" (whatever those mean) that operate on the request's identity attribute set by the logins infrastructure here in darc-mongoreadyserver. Importing this module will scan all custom modules loaded and create a route-data-store for them, storing for each route in the tree thus constructed an integer for the required rights (presuming these are coded into the route spec like method, route, tests etc). Then the allowRequest method simply compares the request's rights (opaquely determined using the roles module) as an integer to the required rights (determined by the route spec) and returns true only if the request's rights are not less than those required.

Adding Tests

Adding Checks

Running

Our intention is that it should be easy to run a server based on this package. To that end we use darc-monngoreadyserverconfig for parsing inputs to the actual run call for your project (e.g., node index.js).

You can specify environment variables, either with export or with a .env file in your project's root directory. All environment variables should be preceeded by your project name, in upper case, with any - characters replaced with _, followed by an _, and then the variable name. For example,

MYPROJECT_CUSTOM_ENV_VAR="Hi!"

This will be read into the configuration object with the key custom-env-var. Of course, the variables can also be the defaults required by the package, as in

MYPROJECT_PORT=6000
MYPROJECT_SSL_CERT=/path/to/sssl/cert
MYPROJECT_SSL_KEY=/path/to/ssl/key

You can specify a configuration file, currently in JSON form. You can set an environment variable MYPROJECT_CONFFILE to point to this file, or you can use the command line argument '--conffile'. Such a configuration file will be read and imported into the configuration directly.

You can also run with long-form command line arguments (run with --help to see which), and you can even specify your own command line arguments (and value checks). Our examples here show how to use command line arguments; to add custom arguments you need to pass arguments optargs and/or checks to the routine returned from the require call:

require( 'darc-monngoreadyserver' )( modules , optargs , checks )
	...

optargs should be an object whose keys are the long-form options to include (minus the -- prefix), and whose corresponding values should be argparse-ready option setting objects. For example,

let optargs = {
	'custom-flag' : { help : 'An extra flag' , action : "storeTrue" , default : false } , 
	'custom-arg' : { help : 'An extra argument' , default : 7 } , 
}

See the argparse docs for more details. checks, if provided, should be an object whose keys are the argument names (e.g., custom-flag) and whose values are strings that can be executed as an eval statement over a single variable val returning a boolean. For example,

let checks = { 
	'custom-arg' : 'val >= 2'
}

These different routes have a specific precedence order: environment variables are overwritten by config file values which are overwritten by command line arguments (to the degree there are any conflicts). The only exception is a config file path itself, which is set to any value from the environment overwritten by any command line argument provided.