@samsystem/winston-elasticsearch v1.1.0-beta.4
winston-elasticsearch
An elasticsearch transport for the winston logging toolkit.
Features
- logstash compatible message structure.
- Thus consumable with kibana.
- Date pattern based index names.
- Custom transformer function to transform logged data into a different message structure.
Compatibility
For Elasticsearch 5.0 and later, use the 0.5.x
series.
For earlier versions, use the 0.4.x
series.
Unsupported / Todo
- Querying.
- Real buffering of messages in case of unavailable ES.
Installation
npm install --save winston winston-elasticsearch
Usage
var winston = require('winston');
var Elasticsearch = require('winston-elasticsearch');
var esTransportOpts = {
level: 'info'
};
winston.add(winston.transports.Elasticsearch, esTransportOpts);
// - or -
var logger = new winston.Logger({
transports: [
new Elasticsearch(esTransportOpts)
]
});
Options
level
info
Messages logged with a severity greater or equal to the given one are logged to ES; others are discarded.index
none the index to be used. This option is mutually exclusive withindexPrefix
.indexPrefix
logs
the prefix to use to generate the index name according to the pattern<indexPrefix>-<indexSuffixPattern>
.indexSuffixPattern
YYYY.MM.DD
a Moment.js compatible date/ time pattern.messageType
log
the type (path segment after the index path) under which the messages are stored under the index.transformer
see below a transformer function to transform logged data into a different message structure.ensureMappingTemplate
true
If set totrue
, the givenmappingTemplate
is checked/ uploaded to ES when the module is sending the fist log message to make sure the log messages are mapped in a sensible manner.mappingTemplate
see fileindex-template-mapping.json
file the mapping template to be ensured as parsed JSON.flushInterval
2000
distance between bulk writes in ms.client
An elasticsearch client instance. If given, all following options are ignored.clientOpts
An object hash passed to the ES client. See its docs for supported options.waitForActiveShards
1
Sets the number of shard copies that must be active before proceeding with the bulk operation.pipeline
none Sets the pipeline id to pre-process incoming documents with. See the bulk API docs.
Important
When changing the indexPrefix
and/ or the transformer
,
make sure to provide a matching mappingTemplate
.
Transformer
The transformer function allows to transform the log data structure as provided by winston into a structure more appropriate for indexing in ES.
The default transformer function's transformation is shown below.
Input:
{
"message": "Some message",
"level": "info",
"meta": {
"method": "GET",
"url": "/sitemap.xml",
...
}
}
}
Output:
{
"@timestamp": "2017-09-30T05:09:08.282Z",
"message": "Some message",
"severity": "info",
"fields": {
"method": "GET",
"url": "/sitemap.xml",
...
}
}
The @timestamp
is generated in the transformer.
Note that in current logstash versions, the only "standard fields" are @timestamp and @version,
anything else ist just free.
A custom transformer function can be provided in the options hash.
Events
error
: in case of any error.
Example
An example assuming default settings.
Log Action
logger.info('Some message', <req meta data>);
Generated Message
The log message generated by this module has the following structure:
{
"@timestamp": "2017-09-30T05:09:08.282Z",
"message": "Some log message",
"severity": "info",
"fields": {
"method": "GET",
"url": "/sitemap.xml",
"headers": {
"host": "www.example.com",
"user-agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"accept": "*/*",
"accept-encoding": "gzip,deflate",
"from": "googlebot(at)googlebot.com",
"if-modified-since": "Tue, 30 Sep 2017 11:34:56 GMT",
"x-forwarded-for": "66.249.78.19"
}
}
}
Target Index
This message would be POSTed to the following endpoint:
http://localhost:9200/logs-2017.09.30/log/
So the default mapping uses an index pattern logs-*
.