Mqtt-meta NPM | npm.io

mqtt-meta

This is a NoSQL Database, to be precise: a JSON Document Store with MQTT interface, CouchDB/MapReduce inspired Views, implemented in Node.js. (Yeah, Buzzword Bingo! 🤠)

It's intended to be used as a database for storing metadata for systems that use MQTT as message bus, I'm using it in conjunction with mqtt-smarthome, but I think it could be useful in other MQTT based environments also.

You can create/modify/delete documents by publishing JSON payload to MQTT and receive document changes by simply subscribing to certain topics. You can create views by defining map and reduce functions in javascript and filter documents with MQTT wildcards.

Install

$ npm install -g mqtt-meta

Usage

Command Line Parameters

Usage: mqtt-meta [options]

Options:
  -v, --verbosity      possible values: "error", "warn", "info", "debug"
                                                               [default: "info"]
  -n, --name           instance name. used as mqtt client id and as prefix for
                       connected topic                         [default: "meta"]
  -u, --url            mqtt broker url.            [default: "mqtt://127.0.0.1"]
  -p, --web-port       web server port                           [default: 8092]
  -i, --web-interface  web server interface                 [default: "0.0.0.0"]
  -w, --web-disable    diable web server
  -h, --help           Show help                                       [boolean]
  --version            Show version number                             [boolean]

You can also connect to the MQTT Broker via Websocket and use TLS, also username ans password is supported for the --url parameter. E.g mqtts://user:password@192.168.2.200:1884 for a MQTT over TLS connection. For Websockets use ws:// respectively wss:// as protocol.

To run mqtt-meta in background and start it on system boot I suggest to use the process manager PM2.

Web UI

mqtt-meta offers a simple ui, you can reach it on http://<hostname>:8092.

Ids

The id of a document can be any string, slashes are allowed - only the mqtt wildcards # and + may not occur in the id. As the intended use of mqtt-meta is to manage metadata that belongs to MQTT topics I suggest to just use the topic that is described by the document as id.

Create/overwrite an object

To create or overwrite an Object with the id hue/light/livingroom you have to publish on the topic meta/set/hue/light/livingroom. The payload has to be a JSON object, e.g. {"type": "light", "name": "hue light livingroom"}. As soon as the document was created the document itself is published retained on the topic meta/doc/hue/lights/livingroom.

Deletion of objects

To delete the object from the previous example just publish on meta/del/hue/lights/livingroom. Payload is irrelevant.

Views

You can create views that publish an array of objects that match certain criteria by publishing a map function, an optional reduce function and an optional filter to the meta/query/<view-id> topic. The map and reduce functions can be any valid Javascript code, if you want to add something to the view you just have to emit it. In the map function this refers to the document, the function is then applied to all documents to compose the view. After the map function completed the optional reduce function is called and can work on the result.

This is loosely inspired by CouchDB and MapReduce, but this is not a "real" MapReduce implementation.

Example: Publish the payload {"map": "if (this.type === 'light') emit(this._id)"} on the topic meta/query/lights to create a view "lights" that contains the ids of all objects that have a type property with the value light. As soon as the view is created mqtt-meta publishes on meta/view/lights, with the above example this would result in following payload:

{
  "result": ["hue/light/livingroom"],
  "length": 1,
  "_rev": 0
}

If any change on the database happens all views are re-composed, so if you add another object with "type": "light" mqtt-meta will immediately re-compose all views and then publish the updated views with the new member.

Besides the possibility to select objects with a map script you can also use the property filter to match documents ids to an mqtt-style wildcard. Example payload: {"filter": "hue/lights/#", "map": "if (this.type === 'color light') return this._id"}

The views are composed in separate worker processes, mqtt-meta will spawn as many workers as CPU cores are available. The map and reduce scripts are executed in a minimal sandbox, so you don't have access to Node.js globals like e.g. console or require. The documents in the workers are frozen, so no change on the database contents is possible by the map and reduce scripts.

See the Wiki for more view examples

Internal properties

These properties are set on all objects by mqtt-meta, they can't be changed or deleted.

`_id`

A documents id.

`_rev`

A documents revision. Just a counter that gets incremented on every change of the object.

Topics on which mqtt-meta publishes

`meta/connected`

Publishes 1 when mqtt-meta is started and publishes 2 when all documents are published.

`meta/rev`

Publishes the database revision number.

`meta/doc/<id>`

All documents are published retained on these topics.

`meta/view/<id>`

All views are published retained on these topics.

Topics subscribed by mqtt-meta

`meta/set/<id>`

Create or overwrite a document.

`meta/extend/<id>`

Extend a document (overwrite only given properties of the document).

`meta/del/<id>`

Delete a document. Payload is irrelevant.

`meta/prop/<id>`

Set/create/delete document properties. Examples Payloads:

{"method":"set", "prop": "name", "val": "new name!"}
{"method":"create", "prop": "name", "val": "new name!"}
{"method":"del", "prop": "name"}

You can use dot-Notation for prop to access nested properties.

In contrast to the set method create won't overwrite existing properties.

`meta/query/<id>`

Create, overwrite or delete a view. Use an empty string payload to delete a view.

Contributing

Pull Requests Welcome! 😀

Disclaimer

I'm not a database expert nor do I think that mqtt-meta as of today scales very well. For my usecase it works with sufficient performance, your mileage may vary.