simpledb-stream v0.0.0
simpledb-stream
Stream the contents of a SimpleDB domain
Requires a configured instance of an aws-sdk SimpleDB
and you get back a Node Streams2 object (based on through2).
Example
var simpledbStream = require('simpledb-stream')
var AWS = require('aws-sdk')
AWS.config.update({
"accessKeyId" : "access key"
, "secretAccessKey" : "secret key"
, "region" : "ap-southeast-2"
})
var simpledb = new AWS.SimpleDB()
simpledbStream(simpledb, 'MyDataDomain')
.on('data', function (data) {
console.log(data)
})
You'll end up with data something like this for each item in your domain:
{
"key": "1ab07b19-6ebb-11e3-b3e2-c321aed38186",
"value": {
"Date": "2013-12-27T05:52:43.696Z",
"RandomNumber": "0.8521310975775123"
}
}
So 'Name'
is converted to 'key'
and 'Attributes'
are squashed into a standard JavaScript object with keys and values representing the 'Name'
and 'Value'
pairs. Use through2-map to adjust the items if you need the output to be a different shape.
Options
chunkSize
simpledbStream(simpledb, { domain: 'MyDataDomain', chunkSize: 100 })
By default simpledb-stream will try for the maximum chunk size it can get from SimpleDB. This can mean larger buffering and longer pauses. If you are close to a SimpleDB end-point then you can shorten the chunk size to stream more efficiently.
criteria
simpledbStream(simpledb, { domain: 'MyDataDomain', criteria: "Date != '0' and Date > '2013-12-27T05:59:11.510Z'"})
Use a 'criteria'
to limit your result set. The criteria will be appended to a 'where ...'
.
Tests
To execute the tests you need a test-aws-config.json file with your AWS credentials. The test suite will create a SimpleDB domain, test the streaming, then delete the test domain.
The test-aws-config.json file should look something like this:
{
"accessKeyId" : "access key"
, "secretAccessKey" : "secret key"
, "region" : "ap-southeast-2"
}
Alternatively this data can be placed, stringified, in a AWS_CONFIG
environment variable prior to test execution.
License
simpledb-stream is Copyright (c) 2013 Rod Vagg @rvagg and licenced under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.
10 years ago