appmetrics-elk v1.0.2

ELK Connector for Node Application Metrics

A connector that collects data using 'appmetrics' and sends it to a configured ElasticSearch instance in LogStash format for use with Kibana.

Getting Started

Installation

The ELK Connector for Node Application Metrics can be installed via npm:

$ npm install appmetrics-elk

This is designed to be used with an ElasticSearch database, and a visualization tool such as Kibana.

Configuring the ELK Connector for Node Application Metrics

The connector can be used in your application by requiring it as the first line of your application:

var appmetrics = require('appmetrics-elk').monitor();

This will send all of the available appmetrics data to the ElasticSearch instance, as well as returning an appmetrics object that can be used to control data collection.

var appmetrics = require('appmetrics-elk').monitor();
appmetrics.disable('mysql');            // disable MySQL monitoring

Additionally, the monitor() API call can be passed an optional ElasticSearch Configuration object to configure the ElasticSearch connection, including database location and security.

The same configuration object can be used to pass configuration to the ELK connector. The following configurations can be applied:

• index (String) the name of the index to use for storing the monitoring data. The default is appmetrics.
• applicationName (String) the name to use for the applicationName field in the monitoring data. The default is the name of the applications main file, eg. app.js.

var config = {
    hosts: [
        'https://es1.bluemix.net',
        'https://es2.bluemix.net'
    ],
    ssl: {
        ca: fs.readFileSync('./cacert.pem'),
        rejectUnauthorized: true
    },
    index: 'nodedata',
    applicationName: 'HelloWorld'
}

var appmetrics = require('appmetrics-elk').monitor(config);
appmetrics.disable('mysql');            // disable MySQL monitoring

Data Provided to ElasticSearch

The ELK Connector for Node Application Metrics uploads its data to the 'appmetrics' index in ElasticSearch. It sends the following values to ElasticSearch for every monitoring entry:

Additional data is then included depending on the monitoring event.

CPU Data

Memory Data

Garbage Collection Data

Event Loop Latency Data

HTTP Request Data

Socket.io Data

MongoDB Query Data

MySQL Query Data

PostgreSQL Query Data

Redis Command Data

Memcached Operation Data

Leveldown Query Data

MQTT Message Data

MQLight Message Data

Sending Custom Data to ElasticSearch

The Node Application Metrics to ELK Connector registers for events that it is aware of, and forwards the data from those events to ElasticSearch. The registration for those events is based on the 'mappings' files in the following directory:

node_modules/appmetrics-elk/mappings/

Any mappings files found in that directory are both used to configure how ElasticSearch handles the data, and to configure the monitoring events that are forwarded.

The type field is used to determine the name of the event to register for, and the properties fields are used to determine the values to send. Note that the values in the properties entry in the mapping file must match the fields in the monitoring event data. For example, the CPU event has the following data:

• process
• system

The mapping file that causes this data to be sent to ElasticSearch therefore has the following structure:

{
    "index":  "appmetrics",
    "type":   "cpu",
    "body": {
        "_source" : {"compress" : true},
        "_ttl" : {"enabled" : true, "default" : "90d"},
        "properties": {
            "timestamp":    {"type": "date", "format": "dateOptionalTime"},
            "hostName":     {"type": "string", "index": "not_analyzed"},
            "pid":          {"type": "integer"},
            "cpu": {
                "type": "nested",
                "include_in_parent": true,
                "properties": {            
                    "process":      {"type": "float"},
                    "system":       {"type": "float"}
                }
            }
        }   
    }
}

This causes the Node Application Metrics to ELK Connector to register for cpu events and forward the process and system values as cpu.process and cpu.system.

Using Kibana with the ELK Connector for Node Application Metrics

During startup the ELK Connector for Node Application Metrics attempts to provide some pre-configuration for using Kibana 4 with the provided data. It does this by uploading the following if there are not existing ones already associated with the 'appmetrics' index:

• An index pattern
• Data mappings for the data types
• Default visualizations for the data types
• A default dashboard

Each of these configurations are dynamically loaded from the 'indexes', 'mappings', 'charts' and 'dashboards' directories in the appmetrics-elk install directory. It is therefore possible to prevent the configurations from being automatically added by deleting those files, or to add to them by adding existing files.

Note: The 'mappings' directory also provides the configuration of which types of monitoring data are uploaded to ElasticSearch so entries should only be deleted if necessary. See Sending Custom Data to ElasticSeach for more information.

Visualizing the data with Kibana 4

The pre-configuration for Kibana 4 provdes a number of default visualizations, as well as a default dashboard. These can subsequently be modified or new visualizations and dashboards created.

Using the dashboard
In order to avoid replacing any dashboard already in use in Kibana 4 with the one supplied by the Node Application Metrics to ELK Connector, the dashboard is made available but not loaded. The dashboard is loaded by:
1. Click on the "Dashboard" tab
2. Select the "Load Saved Dashboard" icon
3. Select "Default AppMetrics Dashboard" from the list of saved dashboards
This now loads a simple dashboard that uses some of the default visualization charts provided by the Node Application Metrics to ELK Connector.

Using the visualization charts
In addition to the dashboard, a number of visualization charts are also provided. These can be added to a dashboard using the following steps:
1. Click on the "Dashboard" tab
2. Click on the "Add Visualization" icon
3. Select a visualization chart from the menu
4. Place and resize the visualization chart by dragging it across the screen

You can also create your own charts using the "Visualize" tab.

License

The Node Application Metrics to ELK Connector is licensed using an Apache v2.0 License.

Version

The current version is 1.0.2

1.0.2 Support for new monitoring data
1.0.1 Support for configurable indexes and addition of applicationName field
1.0.0 Initial release