oss-streamlogger v1.0.1

oss-streamlogger

A Writable Stream object that uploads to s3 objects, periodically rotating to a new object name.

See also tails3 for a script to tail the log files produced by s3-streamlogger.

Installation

npm install oss-streamlogger

Basic Usage

var OssStreamLogger = require('oss-streamlogger').OssStreamLogger;

var oss_stream = new OssStreamLogger({
             bucket: "mys3bucket",
      access_key_id: "...",
  secret_access_key: "..."
});

oss_stream.write("hello S3");

Use with Winston: Log to S3

npm install --save winston
npm install --save oss-streamlogger

var winston        = require('winston');
var OssStreamLogger = require('oss-streamlogger').OssStreamLogger;

var oss_stream = new OssStreamLogger({
             bucket: "mys3bucket",
      access_key_id: "...",
  secret_access_key: "..."
});

var logger = new (winston.Logger)({
  transports: [
    new (winston.transports.File)({
      stream: oss_stream
    })
  ]
});

logger.info('Hello Winston!');

Define subfolder

var OssStreamLogger = require('s3-streamlogger-daily').OssStreamLogger;

var oss_stream = new OssStreamLogger({
             bucket: "mys3bucket",
             folder: "my/nested/subfolder",
      access_key_id: "...",
  secret_access_key: "..."
});

oss_stream.write("hello S3");

Add hostname information for tails3

tails3 expects messages to be logged as json (the default for the file transport), with hostname and (for critical errors), stack properties to each log object, in addition to the standard timestamp, level and message properties. You can provide these using the third "metadata" option to winston's log method:

logger.log(level, message, {hostname: ... , stack: ...});

Handling logging errors

When there is an error writing to s3, the stream emits an 'error' event with details. You should take care not to log these errors back to the same stream (as that is likely to cause infinite recursion). Instead log them to the console, to a file, or to SNS using winston-sns.

Note that these errors will result in uncaught exceptions unless you have an error event handler registered, for example:

oss_stream.on('error', function(err){
    // there was an error!
    some_other_logging_transport.log('error', 'logging transport error', err)
});

Options

bucket (required)

Name of the oss bucket to upload data to. Must exist. Can also be provided as the environment variable BUCKET_NAME.

endpoint (required)

OSS endpoint must be following the region of the bucket. The full list of options is available on the OSS Regions and endpoints page.

access_key_id (required)

OSS access key ID, must have put permission on the specified bucket.

secret_access_key (required)

OSS secret key for the access_key_id specified.

config

Configuration object for the OSS SDK. The full list of options is available on the OSS Parameters page. This is an alternative to using access_key_id and secret_access_key and is overwritten by them if both are used.

folder

An optional folder to stream log files to. Takes a path string, eg: "my/subfolder" or "nested".

name_format

Format of file names to create, accepts strftime specifiers. Defaults to "%Y-%m-%d-%H-%M-%S-%L-unknown-unknown.log". The Date() used to fill the format specifiers is created with the current UTC time, but still has the current timezone, so any specifiers that perform timezone conversion will return incorrect dates.

If you use a format of the form %Y-%m-%d-%H-%M-<stage>-<hostname>.log, then you can use tails3 to tail the log files being generated by OssStreamLogger.

rotate_every

Files will be rotated every rotate_every milliseconds. Defaults to 3600000 (60 minutes).

This fork also specifically supports daily rotation. rotate_every can either be an integer, or a string "day".

max_file_size

Files will be rotated when they reach max_file_size bytes. Defaults to 200 KB.

upload_every

Files will be uploaded every upload_every milliseconds. Defaults to 20 seconds.

buffer_size

Files will be uploaded if the un-uploaded data exceeds buffer_size bytes. Defaults to 10 KB.

License

ISC: equivalent to 2-clause BSD.