reliable-get v5.0.10

Reliable HTTP get wrapper (with cache and serve stale on error), best wrapped around things you dont trust very much.

Behaviour

If reliable-get makes a request that times out or errors, its callback will receive both an error object and a previously cached response, if one is present in cache. You can then decide whether to ignore the error and use the cached response, or not.

Basic usage

var ReliableGet = require('reliable-get');
var config = {
  cache:{
    engine:'memorycache'
  }
};
var rg = new ReliableGet(config);
rg.on('log', function(level, message, data) {
  // Wire up to your favourite logger
});
rg.on('stat', function(type, key, value) {
 // Wire up to your favourite stats library (e.g. statsd)
});
rg.get({url:'http://www.google.com'}, function(err, response) {
   console.log(response.content);
});

Configuration options

When you create an instance of reliable-get you need to specify the cache configuration. This then applies across all requests.

var config = {
  cache:{
    engine:'redis',
    url:'redis://localhost:6379?db=0'
  }
};

You can also pass a property requestOpts to pass options to be used in request. Example:

var config = {
  cache: {
    engine: 'redis',
    url: 'redis://localhost:6379?db=0'
  },
  requestOpts: {
    forever: true,
    followRedirect: false
  }
}

GET options

When making a get request, you need to provide a basic options object:

rg.get({url:'http://www.google.com'}, function(err, response) {
   console.log(response.content);
});

Example from a Compoxure backend request:

var options = {
  url: targetUrl,
  cacheKey: targetCacheKey,
  cacheTTL: targetCacheTTL,
  timeout: utils.timeToMillis(backend.timeout || DEFAULT_LOW_TIMEOUT),
  headers: backendHeaders,
  tracer: req.tracer,
  statsdKey: 'backend_' + utils.urlToCacheKey(host)
};

From a compoxure fragment request:

var options = {
  url: url,
  timeout: timeout,
  cacheKey: cacheKey,
  cacheTTL: cacheTTL,
  explicitNoCache: explicitNoCache,
  headers: optionsHeaders,
  tracer: req.tracer,
  statsdKey: statsdKey
};

The options object is fully passed down to the request.

Response fields

The library will decorate with response with some useful keys that you may need to use, there are following:

• stale - is added to response when the request to origin failed and a stale cached version is returned instead
• cached - is true if there was a cache hit, otherwise false
• realTiming - show the time it took for the response to be returned

Tags

Some store (memorycache/redis) supports assigning one or more tags to a certain resource. They are used only if the resource is cached, to purge all cache entries with the same tag.

Configuration

Cache configuration

The cache object accept any config value accepted by redis. It also takes:

• config.cache.engine: nocache, memorycache, memcached, redis (use nocache/memorycache for testing only!)
• config.cache.url: redis dsn, it is translated to the connection parameters
• config.cache.compress: enable snappy compression on cached items
• config.cache.namespace: adds this string as a prefix to any key. Useful to share redis with other services or migrations
• config.cache.hosts: Memcached cluster nodes addresses as <host>:<port> combinartions in an array
• config.cache.autodiscover: enable AWS Elasticache Auto Discovery of Memcached cache cluster nodes

Request Configuration

The "config.requestOpts" contains the default configuration passed to "request".