Okanjo-app-mongo NPM

Okanjo MongoDB Service

Service for interfacing with MongoDB for the Okanjo App ecosystem.

This package:

Uses Mongoose for object modeling
Manages connectivity and reconnection edge cases
Can manage multiple schemas, even on separate hosts
Provides a reusable CRUD service, useful for extending model services
Packs a bunch of useful utility functions

Installing

Add to your project like so:

npm install okanjo-app-mongo

Note: requires the okanjo-app module.

Breaking Changes

v2.0.0

CrudService: all callback functions return promises. Callbacks are optional.
CrudService._find no longer returns a query. use CrudService._buildQuery instead.
CrudService._createWithRetry signature has changed (is no longer recursive)
CrudService._update no longer returns the isModified flag
CrudService._delete no longer returns the isModified flag

Example Usage

Here's an example app:

example-app
- schemas/
  - widgets.js
- services/
  - DoodadService.js
- config.js
- index.js

`example-app/schemas/widgets.js`

This file contains the Mongoose model definitions and exports a function for MongoService to use when connecting.

"use strict";

const Mongoose = require('mongoose');
const schema = {};


schema.doodad = new Mongoose.Schema({
    name: String,
    key: { type: String, index: { unique: true, dropDups: true } },

    status: { type: String, index: true },
    owner: { type: Mongoose.Schema.Types.ObjectId, index: true },

    created: { type: Date, default: Date.now, index: true },
    updated: { type: Date, default: null }
});


module.exports = function(connection/*, app*/) {
    return {
        Doodad: connection.model('doodad', schema.doodad)
    }
};

`example-app/services/DoodadService.js`

This is an example of using CrudService as a base for model services. You could add your business logic and other model-related functions to this class.

"use strict";

const CrudService = require('okanjo-app-mongo/CrudService');
const BaseId = require('base-id');

/**
 * Doodad Service Example
 */
class DoodadService extends CrudService {

    /**
     * Doodad constructor
     * @param {OkanjoApp} app -
     */
    constructor(app) {
        super(app, null);

        this._modifiableKeys = ['name'];
        this._concealDeadResources = false;

        app.once('ready', () => this.model = app.dbs.widgets.Doodad);
    }


    /**
     * Generates an new Doodad key string
     * @return {String}
     */
    generateKey() {
        return BaseId.base62.generateToken(10, "doodad_" + this.app.dbs.getEnvironmentIdPrefix());
    }


    // Expose and wrap CRUD functions ----


    /**
     * Creates a new Key model
     * @param {*} data – Model properties
     * @param {function(err:Error,obj:Model)} callback - Callback to fire when complete
     */
    create(data, callback) {
        // keys are unique so use _createWithRetry instead of _create
        this._createWithRetry(data, (data) => {
            return {
                name: data.name || "",
                key: this.generateKey(),
                status: DoodadService.status.active,
                account_id: data.account_id
            };
        }, callback);
    }


    /**
     * Retrieves a Key given an identifier
     * @param {ObjectId|string} id - ObjectId or convertible identifier
     * @param {function(err:Error, doc:Model)} [callback] – Fired when completed
     */
    retrieve(id, callback) {
        this._retrieve(id, callback);
    }


    /**
     * Retrieves one or more keys that match the given criteria
     * @param {*} criteria - Filter criteria
     * @param {{[skip]:number, [take]:number, [fields]:string|*, [sort]:*, [exec]:boolean}} [options] - Query options
     * @param {function(err:Error, docs:[Model])} [callback] – Fired when completed
     * @return {Query}
     */
    find(criteria, options, callback) {
        this._find(criteria, options, callback);
    }



    /**
     * Updates a Key model
     * @param key - Key to update
     * @param {*} [data] - Data to apply to the model before saving
     * @param {function(err:Error, obj:Model)} [callback] – Fired when saved or failed to save
     */
    update(key, data, callback) {
        this._update(key, data, callback);
    }


    /**
     * Deletes a key model (make dead)
     * @param key - Key to update
     * @param {function(err:Error, obj:Model)} [callback] – Fired when saved or failed to save
     */
    delete(key, callback) {
        this._delete(key, callback);
    }


    /**
     * Formats a Key or an array of Key models for public consumption
     * @param {[Model]|Model} mixed - Key or array of Keys
     */
    formatForResponse(mixed) {
        return this.app.response.formatForResponse(mixed, (obj) => {
            return {
                id: this.app.mongo.getPublicId(obj._id, this.app.prefixes.doodad),
                name: obj.name,
                key: obj.key
            };
        });
    }
}

/**
 * Status enumeration
 * @type {{dead: string, active: string}}
 */
DoodadService.status = {
    dead: "dead",
    active: "active"
};


module.exports = DoodadService;

`example-app/config.js`

Typical OkanjoApp configuration file, containing the mongo config

"use strict";

const Path = require('path');

// this is just for making running the example across platforms easy
// you generally won't need this line
const host = process.env.MONGO_HOST || '192.168.99.100:9010';

module.exports = {
    mongo: {
        schemas: [
            {
                name: 'widgets',
                path: Path.join(__dirname, 'schemas', 'widgets.js'),
                uri: `mongodb://${host}/unittest_widgets`
            }
        ]
    }
};

`index.js`

Example application that will connect, create a doc, and find the doc.

"use strict";

const OkanjoApp = require('okanjo-app');
// const MongoService = require('okanjo-app-mongo');
const MongoService = require('../../MongoService');

const DoodadService = require('./services/DoodadService');
const config = require('./config');

// Init the app
const app = new OkanjoApp(config);

// Add the mongo service to the app
app.dbs = new MongoService(app, config.mongo);

// Maybe you want to use CrudService for building model-based services
app.services = {
    doodad: new DoodadService(app)
};

// Start it up
app.connectToServices(() => {

    // Example: use the extended CrudService to make a new doc
    app.services.doodad.create({
        name: 'my doodad',
        account_id: 'user_1'
    }, (err, doc) => {
        if (err) {
            app.report('Blew up creating new doodad', err);
            process.exit(1);
        } else {

            app.dump('Created doodad', doc.toObject());

            // Example: use the direct Mongoose model to find the doc
            app.dbs.widgets.Doodad.find({ _id: doc._id }, (err, docs) => {
                if (err) {
                    app.report('Blew up retrieving doodads', err);
                    process.exit(2);
                } else {

                    app.dump('Retrieved doodads', docs.map((d) => d.toObject()));

                    console.log('Done!');
                    process.exit(0);
                }
            });

        }
    });
});

A runnable version of this application can be found in docs/example-app.

MongoService

MongoDB management class. Must be instantiated to be used.

Properties

mongo.app – (read-only) The OkanjoApp instance provided when constructed
mongo.config – (read-only) The mongo service configuration provided when constructed
mongo.prefixes – Object identifier prefixes
mongo.prefixAliases – Old Object identifier prefixes that map to a new one
mongo[schemaName] – Each connected schema will load its Mongoose models here. Just don't name your schema the same as any class property or method!

Methods

`new MongoService(app, [config])`

Creates a new mongo service instance.

app – The OkanjoApp instance to bind to
config – (Optional) The mongo service configuration object. Defaults to app.config.mongo if not provided.
- config.prefixes – Optional mappings for pretty ids, where the key maps to the prefix. E.g. { "thing": "tng", "product", "pr" }
- config.prefixAliases – Optional aliases for mappings, where the key matches the prefixes key and the value is the old or aliased prefix. Useful for migrating from an old id prefix scheme to a new one.
- config.schemas – Optional array of schema connections.
  - config.schemas[].name – Required reference name of the schema. It will be added as a property of the class when connected. E.g. "widgets"
  - config.schemas[].uri – Required connection URI for mongodb. E.g. mongodb://host:port/databasename
  - config.schemas[].path – Required string path that exports a function which returns the models built on the connection. function(connection, app) { return { Model: connection.model('doodad', ...) }; }

`mongo.getObjectId(mixed_id)`

Returns an ObjectId from a given identifier.

mixed_id – An identifier. Can be an ObjectId or a string. If given as a string, the string may a 12-byte hexadecimal value or be a prefixed base-58 encoded value.

`mongo.compareIds(mixed_id, mixed_id)`

Returns true if the two identifiers match.

mixed_id – An identifier. Can be an ObjectId or a string. If given as a string, the string may a 12-byte hexadecimal value or be a prefixed base-58 encoded value.

`mongo.isModel(obj)`

Returns whether the given object is a Mongoose model or not.

obj – An object to test

`mongo.getEnvironmentIdPrefix()`

Returns the current app environment to prefix to an id. The following environment names are handled specially:

default – This is assumed to be running locally, so the prefix will be local_
production – This is assumed to be running in a live environment, so there will be no prefix.

Anything else will be returned as is. For example, if the current app environment is sandbox then sandbox_ will return.

`mongo.getPublicId(id, prefix)`

Returns a public readable id for an ObjectId. Useful for making public identifiers readable and not entirely gibberish or generic hex values. Values will be returned with the given prefix and encoded in base-58.

id – ObjectId to format
prefix – Prefix describing what the identifier is for. For example, product.

Note: The environment prefix will be included as well. For example, in sandbox environment, the returned value might look like product_sandox_asdfasdfasdf.

Events

`mongo.on('health_change', (newState) => { ... })`

Fired when the mongo service aggregate connection status changes.

newState – Boolean whether all connections are ready or not.

CrudService

Base class for building services based on a Mongoose model. The idea of using CrudService is to:

Stop duplicating logic across every single service you have to write (CRUDL)
Automatically handle and report errors on common operations so you don't need to in the business logic
Provide base functions that can be optionally used in the service when exposed as whatever names you like
- This also allows you to hook-in logic on various events (e.g. when service.delete is called, do something special)
Conceal deleted resources without actually deleting them
- We don't like to permanently delete data. Instead, we like to leave tombstones behind so we can audit before cleaning up later. This is also very handy for syncing to data lakes. Do you know what resources were deleted in the last 15 minutes?
- When a doc is deleted, its status property is just set to dead.
- The _find and _retrieve helpers automatically deal with dead resources from there, like they were really deleted.

Note: you should extend this class to make it useful!

Properties

service.app – (read-only) The OkanjoApp instance provided when constructed
service.model – The Mongoose model this service manages
service._createRetryCount – How many times a _createWithRetry method can attempt to create a doc before giving up
service._modifiableKeys – What model properties are assumed to be safe to copy from user-data
service._deletedStatus – The status to set docs to when "deleting" them
service._concealDeadResources – Whether this service should actively prevent "deleted" (status=dead) resources from returning in _retrieve and _find

Methods

`new CrudService(app, model, dbService)`

Creates a new instance of a crud service

app – The OkanjoApp instance
model – The mMongoose model this service should manage (or leave null to set later)
dbService – The MongoService instance which handles the connection. Defaults to app.dbs

`_create(data, [callback], [suppressCollisionError])`

Creates a new resource.

data – The object to store
callback(err, doc) – Optional, function fired when completed
- err – Error, if occurred
- doc – The new Mongoose model that was created
suppressCollisionError - Internal flag to suppress automatically reporting the error if it is a collision
Returns a Promise

`_createWithRetry(data, objectClosure, [callback])`

Creates a new resource after calling the given object closure. This closure is fired again (up to service._createRetryCount times) in the event there is a collision. This is useful when you store documents that have unique fields (e.g. an API key) that you can regenerate in that super rare instance that you collide

data – The object to store
objectClosure(data, attempt) – Function fired before saving the new document. Set changeable, unique properties here
- data – The object to store
- attempt – The attempt number, starting at 0
callback(err, doc) – Optional, function fired when completed
- err – Error, if occurred
- doc – The new Mongoose model that was created
Returns a Promise

`_retrieve(id, [callback])`

Retrieves a single document from the collection.

id – The mixed id of the record. Can be an ObjectId or public base-58 encoded id
callback(err, doc) – Optional, function fired when completed
- err – Error, if occurred
- doc – The Mongoose model found or null if not found
Returns a Promise

`_find(criteria, [options], [callback])`

Finds records matching the given criteria. Supports pagination, field selection and more!

criteria – Object with mongo query criteria
options – (Optional) Additional query options or mongo query settings
- options.skip – Offsets the result set by this many records (pagination). Default is unset.
- options.take – Returns this many records (pagination). Default is unset.
- options.fields – Returns only the given fields (same syntax as mongo selects) Default is unset.
- options.sort – Sorts the results by the given fields (same syntax as mongo sorts). Default is unset.
- options.conceal – Whether to conceal dead resources. Default is true.
- options.* – Any other option is passed to Mongoose Query#setOptions.
callback(err, docs) – Optional, fired when completed
- err – Error, if occurred
- docs – The array of documents returned or [] if none found.
Returns a Promise

`_count(criteria, [options], [callback])`

Counts the number of matched records.

criteria – Object with mongo query criteria
options – (Optional) Additional query options or mongo query settings
- options.conceal – Whether to conceal dead resources. Default is true.
- options.* – Any other option is passed to Mongoose Query#setOptions.
callback(err, count) – Optional, fired when completed
- err – Error, if occurred
- count – The number of matched documents or 0 if none found.
Returns a Promise

`_update(doc, [data], [callback])`

Updates the given model and optionally applies user-modifiable fields, if service is configured to do so.

doc – The model to update
data – (Optional) Additional pool of key-value fields. Only keys that match service._modifiableKeys will be copied if present. Useful for passing in a request payload and copying over pre-validated data as-is.
callback(err, doc) – Optional, fired when completed
- err – Error, if occurred
- doc – The updated model
Returns a Promise

`_delete(doc, [callback])`

Fake-deletes a model from the collection. In reality, it just sets its status to dead (or whatever the value of service._deletedStatus is).

doc – The model to delete
callback(err, doc) – Optional, fired when completed
- err – Error, if occurred
- doc – The updated model
Returns a Promise

`_deletePermanently(doc, [callback])`

Permanently deletes a model from the collection. This is destructive!

doc – The model to delete
callback(err, doc) – Optional, fFired when completed
- err – Error, if occurred
- doc – The deleted model
Returns a Promise

Events

This class does not emit events.

Extending and Contributing

Our goal is quality-driven development. Please ensure that 100% of the code is covered with testing.

Before contributing pull requests, please ensure that changes are covered with unit tests, and that all are passing.

Testing

Before you can run the tests, you'll need a working mongodb server. We suggest using docker.

For example:

docker pull mongo:4.0
docker run -d -p 27017:27017 mongo:4.0

To run unit tests and code coverage:

MONGO_HOST=localhost:27017 npm run report

Update the MONGO_HOST environment var to match your docker host (e.g. 127.0.0.1, user, pass, etc)

This will perform:

Unit tests
Code coverage report
Code linting

Sometimes, that's overkill to quickly test a quick change. To run just the unit tests:

npm test

or if you have mocha installed globally, you may run mocha test instead.

Okanjo OkanjoApp Mongo MongoDB Mongoose

base-id mongoose

@infinitebrahmanuniverse/nolb-ok @zalastax/nolb-ok @everything-registry/sub-chunk-2372

2 years ago

2 years ago

4 years ago

4 years ago

5 years ago

5 years ago

5 years ago

6 years ago