1.3.4 • Published 4 years ago

@geraint-37d5/logdna-logger-node v1.3.4

Weekly downloads
2
License
MIT
Repository
github
Last release
4 years ago

Coverage Status

Migrating From Other Versions

Previous versions of this client are still supported, but if you are upgrading to this version, please see our migration document for the differences between this version and prior versions.

Install

$ npm install @logdna/logger

Setup

Operation requires a LogDNA Ingestion Key. Without it, the client will not be able to post logs to the cloud. Please contact our support if you have questions about this setup process!

Usage

To use the client, create an instance, then call .log() or a convenience method.

const logdna = require('@logdna/logger')

const options = {
  app: 'myAppName'
, level: 'debug' // set a default for when level is not provided in function calls
}

const logger = logdna.createLogger('<YOUR INGESTION KEY>', options)

logger.log('This is an INFO statement', 'info')

logger.log('This will be a DEBUG statement, based on the default')

logger.log('This is an INFO statement with an options object', {
  level: 'info'
})

logger.info('This is an INFO statement using a convenience method')

logger.error('This is an error with meta data attached', {
  indexMeta: true
, meta: {
    message: 'TypeError for XYZ'
  , err
  }
})

logger.error(err) // Objects can be logged too

Supported Log Levels

The client supports the following log levels. They are case-insensitive.

  • TRACE
  • DEBUG
  • INFO
  • WARN
  • ERROR
  • FATAL

Convenience Methods

We have set up convenience methods that automatically set the log level appropriately, and are easy to read.

logger.trace(msg[, options])

logger.debug(msg[, options])

logger.info(msg[, options])

logger.warn(msg[, options])

logger.error(msg[, options])

logger.fatal(msg[, options])

Events

The Logger is an EventEmitter and will emit events rather than use promises or callbacks to communicate its progress. Listening to the events is optional, although an error listener is recommended.

Event: 'addMetaProperty'

Emitted when a meta property is successfully added. This meta property will be attached to each log message.

Event: 'cleared'

  • <Object>
    • message <String> - A message indicating that everything was sent or that nothing needed to be sent

When all log lines have been sent to LogDNA, this event is emitted. If it emits after lines have successfully been sent, then the message will be 'All accumulated log entries have been sent'. If there were no lines to be sent (for example, if flush() was called proactively), then the message will be 'All buffers clear; Nothing to send'.

Event: 'error'

This event is emitted when an asynchronous error is encountered. Depending on the context, meta will contain different pieces of information about the error.

Error while sending to LogDNA

The metadata for an error encountered during an HTTP call to LogDNA will have the following meta properties in the error:

  • actual <String> - The raw error message from the HTTP client
  • code <String> | <Number> - The HTTP agent's "code" or statusCode value
  • firstLine <String> - The first log line in the buffer that was sending
  • lastLine <String> - The last log line in the buffer that was sending
  • retrying <Boolean> - Whether an attempt will be made to resend the payload
  • attempts <Number> - The number of consecutive failures

Error while calling log()

When log() is called directly, or indirectly (via a convenience method), errors can be emitted if certain validations fail. If an invalid log level is provided, or if a bad data type is found for the options parameter, the meta property of the error will contain the following properties:

  • got <String> - Description of the invalid input. Will depend on error context.
  • expected <String> - The allowable log levels if options is an invalid log level
  • used <String> - If a bad level is used in options, it will be ignored, and the default will be used. This property indicates what that value is.

Error due to payloadStructure mismatch

When log() or agentLog() is called, the payloadStructure must be set appropriately. If it is not, an error is emitted. Keep in mind that agentLog() is reserved for LogDNA systems and is not intended for public usage.

  • message <String> - Static message of Invalid method based on payloadStructure
  • payloadStructure <String> - The current payload structure value that is set on the instance
  • expected <String> - The expected payload structure to be able to call the method.

Error for a 207 (partial success) response

If a 207 status code is received, this means that some of the lines failed to be ingested. An error event is emitted for each failed line and will have the following structure:

  • message <String> - Static message: Non-200 status while ingesting this line
  • meta <Object> - Details about the failed line
    • statusCode <Number> - The http status code for the failed line
    • line <String> - The line that failed to be ingested

Event: 'removeMetaProperty'

This is emitted when a key (and implied value) are removed from the global meta object. If the key does not exist, then a warn event with the same signature will be emitted instead.

Event: 'send'

  • <Object>
    • httpStatus <String> - The status property of the HTTP agent's response
    • firstLine <String> - The first log line in the buffer that was sent
    • lastLine <String> - The last log line in the buffer that was sent
    • totalLinesSent <Number> - The total number of lines in the sent buffer
    • totalLinesReady <Number> - The number of lines left to be sent (if queueing has happened)
    • bufferCount <Number> - The number of buffers left to be sent (if queueing has happened)

This event is emitted when a buffer is successfully sent to LogDNA. Since a buffer can contain many log entries, this event summarizes the activity. In a high throughput system where flushLimit is exceeded and multiple buffers are waiting to be sent, information like totalLinesReady and bufferCount help illustrate how much work is left to be done. Any buffers that have been queued will be sent one after another, ignoring any flush timer.

Event: 'warn'

This event is emitted when there is no log data provided to the log method, or when removeMetaProperty is called with an unrecognized key. For those cases, additional properties (apart from message) are included:

Warnings during log()

  • statement (Any) - If log() was called with a null string or an invalid data type, this key will contain the given log statement.

Warnings during agentLog()

  • statement (Any) - If agentLog() was called with a null string or an invalid data type, this key will contain the given log statement.

Warnings during removeMetaProperty

  • key <String> - The key that the command attempted to remove but that did not exist

API

createLogger(key[, options])

  • key <String> - Your ingestion key
  • options <Object>
    • level <String> - Level to be used if not specified elsewhere. Default: INFO
    • tags <Array> | <String> - Tags to be added to each message
    • meta <Object> - Global metadata. Added to each message, unless overridden.
    • timeout <Number> - Millisecond timeout for each HTTP request. Default: 30000ms. Max: 300000ms
    • hostname <String> - Hostname for each HTTP request.
    • mac <String> - MAC address for each HTTP request.
    • ip <String> - IPv4 or IPv6 address for each HTTP request.
    • url <String> - URL of the logging server. Default: https://logs.logdna.com/logs/ingest
    • flushLimit <Number> - Maximum total line lengths before a flush is forced. Default: 5000000
    • flushIntervalMs <Number> - Mseconds to wait before sending the buffer. Default: 250ms
    • shimProperties <Array> - List of dynamic options keys to look for when calling log()
    • indexMeta <Boolean> - Controls whether meta data for each message is searchable. Default: false
    • app <String> - Arbitrary app name for labeling each message. Default: default
    • env <String> - An environment label attached to each message
    • baseBackoffMs <Number> - Minimum exponential backoff time in milliseconds. Default: 3000ms
    • maxBackoffMs <Number> - Maximum exponential backoff time in milliseconds. Default: 30000ms
    • withCredentials <Boolean> - Passed to the request library to make CORS requests. Default: false
    • payloadStructure <String> - (LogDNA usage only) Ability to specify a different payload structure for ingestion. Default: default
    • compress <Boolean> - (LogDNA usage only) Compression support for the agent. Default: false
    • proxy <String> - The full URL of an http or https proxy to pass through
  • Throws: <TypeError> | <TypeError> | <Error>
  • Returns: Logger

Returns a logging instance to use. flushLimit and flushIntervalMs control when the buffer is sent to LogDNA. The flushIntervalMs timer is only started after lines are logged, and the flushLimit is a size approximation based on the summation of .length properties of each log line. If the buffer size exceeds flushLimit, it will immediately send the buffer and ignore the flushIntervalMs timer. Otherwise, a timer will repeatedly flush the buffer every flushIntervalMs milliseconds, as long as the buffer contains log entries.

If indexMeta is false, then the metadata will still appear in LogDNA search, but the fields themselves will not be searchable. If this option is true, then meta objects will be parsed and searchable up to three levels deep. Any fields deeper than three levels will be stringified and cannot be searched. WARNING: When this option is true, your metadata objects across all types of log messages MUST have consistent types, or the metadata object may not be parsed properly!

shimProperties can be used to set up keys to look for in the options parameter of a log() call. If the specified keys are found in options, their key-values will be included the top-level of the final logging payload send to LogDNA.

payloadStructure is only for LogDNA's use in other parts of the system such as our logging agent. It is not intended to be used by public consumers, and it should be left to the default value.

For more information on the backoff algorithm and the options for it, see the Exponential Backoff Strategy section.

setupDefaultLogger(key[, options])

The same as createLogger(), except for that it creates a singleton that will be reused if called again. Users can call this multiple times, and the client package will maintain (create and/or return) the singleton.

Note that only the first call will instantiate a new instance. Therefore, any successive calls will ignore the provided parameters.

const logdna = require('@logdna/logger')

const logger = logdna.setupDefaultLogger('<YOUR KEY HERE>')
const sameLogger = logdna.setupDefaultLogger()

logger.agentLog(opts)

This method is for use exclusively by LogDNA, and is not intended for public logging.

logger.addMetaProperty(key, value)

This method adds a key-value to the global metadata, which is added to each log entry upon calling log(). Although meta can be set on instantiation, this method provides a way to update it on-the-fly.

If options.meta is also used in a log() call, the key-value pairs from the global meta will be merged with options.meta, and those new pairs will take precedence over any matching keys in the global metadata.

// This will use `meta` to track logs from different modules
const logger = createLogger('<YOUR API KEY>', {
  meta: {
    module: 'main.js' // Global default
  }
})

logger.debug('This is the main module') // Uses global meta

// ... elsewhere, in another file, perhaps
logger.info('I am in module1.js', {
  meta: {module: __filename} // Overrides global meta
})

logger.flush()

When flush is called, any messages in the buffer are sent to LogDNA. It's not necessary to call this manually, although it is useful to do so to ensure clean shutdown (see Best Practices). When log is called, it automatically starts a timer that will call flush, but it is idempotent and can be called at any time.

If log lines exist in the current buffer, it is pushed onto a send queue, and a new buffer is created. The send queue is processed and uploaded to LogDNA.

If no work needs to be done, the cleared event is immediately emitted.

logger.log(statement[, options])

  • statement <String> | <Object> - Text or object of the log entry. Objects are serialized.
  • options <String> | <Object> - A string representing a level or an object with the following elements:
    • level <String> - Desired level for the current message. Default: logger.level
    • app <String> - App name to use for the current message. Default: logger.app
    • env <String> - Environement name to use for the current message. Default: logger.env
    • timestamp <Number> - Epoch ms time to use for the current message. Must be within 24 hours. Default: Date.now()
    • context <Object> - Synonym for meta, but mutually exclusive. Ignored if meta exists.
    • indexMeta <Boolean> - Allows for the meta to be searchable in LogDNA. Default: logger.indexMeta
    • meta <Object> - Per-message meta data. Combined with key-values created with addMetaProperty
  • Emits: warn, error

Sends a string or object to LogDNA for storage. If the convenience methods are used, they call this function under the hood, so the options are the same. The only difference is that level is automatically set in the convenience methods.

logger.removeMetaProperty(key)

  • key <String> - The key (and implied value) to be removed from the global meta object.
  • Emits: warn

Attempts to remove the given key from the global meta data object. If the key is not found, warn is emitted.

How Log Lines are Sent to LogDNA

In default operation, when log functions are called, the line is added to a buffer to cut down on HTTP traffic to the server. The buffer is flushed every flushIntervalMs milliseconds or if the log line lengths grow beyond flushLimit.

When flush fires (or is called manually), the current buffer is put onto a send queue, and a new buffer is started. The send queue begins sending to LogDNA. It will continue to send without pausing or honoring flushIntervalMs as long as there are buffers in the send queue. When the send queue is empty, cleared is emitted.

Handling Errors

  • If a 207 status code was received, this means that at least one line failed ingestion. Each offending line and its status code will be emitted as an error.
  • User-level errors (such as 400) will not be retried because they most likely would never be successful (if the message is deemed invalid), and error events are emitted for these errors, also.
  • Connection-based errors or 500-level response status codes will be retried using an exponential backoff strategy, but will also emit error events along the way.
    • Connection error codes that will retry: ECONNABORTED (timeout), ECONNRESET, EADDRINUSE, ECONNREFUSED, EPIPE, ENOTFOUND, ENETUNREACH
    • HTTP status codes that will retry: 500, 502, 503, 504, 521, 522, 524

Exponential Backoff Strategy

When HTTP failures happen, if they are deemed "retryable" (non-user errors), then the client will pause for a short time before trying to resend. The algorithm it implements is an exponential backoff with a "jitter" strategy that uses random numbers statistically to spread out the wait times to avoid flooding.

The settings for baseBackoffMs and maxBackoffMs are used in this algorithm and serve as the lower and upper boundaries for the wait time.

These types of errors are blocking since they are related to timeouts and server errors. Logs will continue to buffer as normal, and if the HTTP calls becomes successful, they will begin to send immediately, and without pause.

Best Practices

  • The client is optimized for high throughput. Using a single instance is no problem, but multiple instances can be created with the same key if desired.
  • Set up an error listener so that your app is aware of problems. Things like HTTP errors are emitted this way.
  • When shutting down your application, ensure all log entries are cleared. Services like AWS Lambda can buffer log entries, so it might be worthwhile to pause for a short time before exiting like in the following example:
const {createLogger} = require('@logdna/logger')
const {once} = require('events')

const logger = createLogger('<YOUR KEY HERE>')

logger.on('error', console.error)

async function clearLogger() {
  logger.flush()
  await once(logger, 'cleared')
  // Everything clear.  Did Lambda buffer anything?
  await sleep(1000)
  logger.flush()
  await once(logger, 'cleared')
}

Client Side

Browserify Example

const {createLogger} = require('@logdna/logger');
const logger = createLogger('API KEY HERE', {
  hostname:'ClientSideTest'
, app: 'sequence'
, indexMeta: true
})

const date = new Date().toISOString()
const logme = () => {
  for (var i = 0; i < 10; i++) {
    logger.log('Hello LogDNA Test ' + date, {
      meta: {
        sequence: i
      }
    })
  }
}

setInterval(logme, 5000);

If the above snippet is saved as a file main.js, then, with Browserify, you can convert this file to a bundle.js file:

browserify main.js -o bundle.js

The bundle.js file can be included like any other script:

<script src="bundle.js"></script>

When using NodeJS inside a browser, the domain needs to be whitelisted in your LogDNA organization settings.

Bunyan Stream

For Bunyan Stream support, reference our logdna-bunyan module.

Winston Transport

For Winston support, reference our logdna-winston module.

AWS Lambda Support

AWS Lambda allows users to add logging statements to their Lambda Functions. You can choose to setup the logger as shown above, or you can override the console.log and console.error statements. AWS Lambda overrides the console.log, console.error, console.warn, and console.info functions as indicated here, within the scope of the handler (main) function. You can setup an override as follows:

'use strict'

const https = require('https')
const Logger = require('@logdna/logger')

const options = {
  env: 'env'
, app: 'lambda-app'
, hostname: 'lambda-test'
, indexMeta: true
}

const _log = console.log
const _error = console.error

const logger = Logger.setupDefaultLogger('YOUR API KEY', options)

function log(...args) {
  logger.log.apply(args)
  _log.apply(...args)
}

function error(...args) {
  logger.error.apply(args)
  _error.apply(args)
}

/**
 * Pass the data to send as `event.data`, and the request options as
 * `event.options`. For more information see the HTTPS module documentation
 * at https://nodejs.org/api/https.html.
 *
 * Will succeed with the response body.
 */
exports.handler = (event, context, callback) => {
  console.log = log
  console.error = error

  // Your code here
  console.log('How bout normal log')
  console.error('Try an error')

  callback()
}

Proxy Support

The logger supports proxying for situations such as corporate proxies that require traffic to be passed through them before reaching the outside world. For such implementations, use the proxy instantiation option to set the full URL of the proxy. It supports both http and https proxy URLs. Under the hood, the logger uses the https-proxy-agent package for this.

In this example, an http proxy (with credentials) is passed through before reaching LogDNA's secure ingestion endpoint:

const {createLogger} = require('@logdna/logger')

const logger = createLogger(apiKey, {
  proxy: 'http://username:pass@yourproxy.company.com:12345'
, app: 'myapp'
})

logger.info('Happy logging through your proxy!')

License

Copyright © LogDNA, released under an MIT license. See the LICENSE file and https://opensource.org/licenses/MIT

Happy Logging!

logdnalogsloggingwinstonbunyannodejsnodelogdna.comloggerjavascript
agentkeepaliveaxioshttps-proxy-agentjson-stringify-safe
@infinitebrahmanuniverse/nolb-_ger@everything-registry/sub-chunk-358
1.3.4

4 years ago

1.3.3

4 years ago