resource-store v0.1.0
resource-store
resource-store
is a Node.js caching/memoization library with multiple storage
backends.
Usage
Create a ResourceStore
object with a generator callback. Then, request
values from it by calling the get
method and passing JavaScript objects or
primitives as keys. If the given key is not already cached, the generator
function will be called to generate its value.
The generator function receives keys and generates the values associated with them. For a given key, the value it returns should always be the same.
The optional backend
argument determines the type of storage used by the store:
- Omitted - In-memory storage only, using
MemoryBackend
. - String - File-backed storage starting at the given directory, using
FileBackend
. - Object - An object with methods
get
,set
,delete
, andlist
, see Backends below for details.
The extra
parameter passed to the generator function is an object that can be
used to store extra data alongside the value generated from the key. See
Out-of-Band Data below for more information.
var ResourceStore = require('resource-store');
var store = new ResoureStore(backend, function generator(key, extra, cb) {
var value;
// generate value based on key
// generator callback takes parameters (err, value)
cb(null, value);
});
Methods
get(key, cb)
Retrieves a value from the store. If the value for the given key has not been
generated yet, the generator function will be called, otherwise the stored
value will be returned. If multiple calls are made to get
for the given key,
then the generator function will only be called once and all outstanding calls
will return when the generator completes.
The callback to get
takes parameters (err
, value
, extra
):
err
-null
unless an error occurred in the generator function or the storage backend.value
- the value associated with the requested key.extra
- see Out-of-Band Data below.
store.get({ prop1 : 'value1' }, function(err, value, extra) {
// do stuff with the returned value
});
delete(key, cb)
Deletes a value from the store. If the value specified by key
does not exist
then an error will be returned.
store.delete({ prop1 : 'value1' }, function(err) {
// delete succeeded if err is null
});
list(cbEntry, cbDone)
Lists all values cached in the store.
cbEntry
is called once for each entry with parameters err
, key
, value
,
extra
. See Out-of-Band Data below for more information
about the extra
parameter.
cbDone
is called after all entries have been listed with parameters err
,
numEntries
.
store.list(function cbEntry(err, key, value, extra) {
// do something with this value
}, function cbDone(err, numEntries) {
// now go do something else
});
Out-of-Band Data
Some callback functions receive an extra
parameter which is the actual value
sent to the storage backend (the value associated with the given key is stored
at extra.value
). The generator function can set properties on extra
to
store data alongside the cached value. This can be useful for debugging.
The library will also set the following values on extra
(all date/time values
are millisecond-precision Unix timestamps, the result of +new Date
):
key
- the key object associated with this item. Not very useful to consumers, but needed because a backend may not know its own keys.value
- the data value associated with this key.createStarted
- available for all functions, but most useful inget
andlist
.createEnded
- set when the generator function completes; available forget
andlist
.lastRetrieved
- set when a value is generated or retrieved from the storage backend. Available inget
andlist
, but most useful inlist
to allow deleting old data.wasCached
- available inget
and set totrue
if the requested value was already cached (i.e. the generator function was not called).baseFilename
- available in all functions if using theFileBackend
for storage. This is the data filename where the value will be stored, without the.json
extension. This is useful for storing additional binary data alongside the key.
Backends
Storage backends that can be used by this library must store data objects
associated with string keys. Note: ResourceStore
accepts object keys,
but storage backends accept string keys.
Backends must implement the following methods:
get
- return the value associated with a key, and set and remembervalue.lastRetrieved
, or returnfalse
if nothing is stored with that key.set
- set the data value associated with a key.delete
- clear the data value associated with a key.list
- list all data values stored.
Backends can have a shouldCache
property. If it is missing or truthy, then
ResourceStore
will remember values retrieved from the backend when multiple
calls to get
are made concurrently for the same key. Set
backend.shouldCache = false
to disable this behavior if retrieving data from
the backend is effectively instantaneous.
Pseudocode (to avoid releasing Zalgo, all of these methods should be asynchronous in real implementations):
// cb takes parameters (err, value)
backend.get = function(keyString, cb) {
// If the key doesn't have anything stored, return false
if (!(keyString in backend.data)) {
return cb(null, false);
}
// Get the data value associated with the key
var value = ...;
// Set lastRetrieved (and persist it somehow)
value.lastRetrieved = +new Date;
// err is null unless getting the value failed
cb(err, value);
};
// cb takes parameters (err)
backend.set = function(keyString, value, cb) {
// Set the data value associated with the key
backend.data[keyString] = value;
// err is null unless setting the value failed
cb(err);
};
// cb takes parameters (err)
backend.delete = function(keyString, cb) {
// Clear the data value associated with the key
// err is null unless deleting the value failed
cb(err);
};
// cbEntry takes parameters (err, key, value)
// cbDone takes parameters (err, numEntries)
backend.list = function(cbEntry, cbDone) {
// Get the list of keys stored
var keys = Object.keys(backend.data),
numEntries = 0;
// Loop and return values
keys.forEach(function(k) {
var value = backend.data[k];
// Skip values that were removed after obtaining the list of keys
if (typeof value != 'undefined') {
// Passing the key is optional: some backends (like FileBackend)
// don't know their own keys, so pass null instead
// Set value.lastRetrieved if it isn't already set
value.lastRetrieved = backend.lastRetrieved[k];
cbEntry(null, k, value);
numEntries++;
}
});
cbDone(null, numEntries);
};
Except for the list
method, backends do not need to worry about concurrency:
ResourceStore
will serialize operations for a given key so that only one of
get
, set
, and delete
is operating at a time. The list
method should
store a list of keys first, then loop over them and skip keys that no longer
have associated data values.
The library comes with two backends already implemented:
ResourceStore.FileBackend
and ResourceStore.MemoryBackend
.
new FileBackend(dir)
stores data in JSON filenames named based on the hash of
the requested keys, starting at the directory dir
and creating subdirectories
as needed.
new MemoryBackend()
stores data in memory in a plain old JavaScript object.
Mutability
When using the MemoryBackend
, values stored in a ResourceStore
are mutable
by default. This has the following consequences:
- The value returned by the generator
===
the value returned toget()
andlist()
for the same key - Object prototype chains, function properties, etc. are preserved
- The
ResourceStore
can be used to cache database connections or other complex objects with internal state.
If you'd rather have an immutable data store instead, construct the
ResourceStore
using a FileBackend
, or using an immutable MemoryBackend
as
follows:
var store = new ResourceStore(
new ResourceStore.MemoryBackend({ mutable : false }),
function generator(key, extra, cb) { ... });
With an immutable backend, the ResourceStore
will only be able to store
values that can be represented in JSON.