oddity v0.0.9
oddity
A streaming Open Data toolkit
##What is Oddity ? Oddity provides tool for extracting and formatting Open Data; it is based on streams.
##How to use it ?
###Installation
npm install oddity
###Sample Oddfile.js
for CSV data
var odd = require('oddity');
var csv = odd.csv;
fromFile('examples/colors.csv')
.pipe(csv.splitLines())
.pipe(csv.toArray())
.pipe(csv.zipHeader())
.pipe(toString())
.pipe(toFile('output.json'))
###Execution
node Oddfile.js
Or if you also intalled oddity globally, just go to the folder containing your Oddfile.js
:
oddity
##Documentation
###Oddity
####.fromFile
Returns a readStream from an input file.
.fromFile(input)
Parameters:
input
: the name of the input file.
Alias: fromFile
####.toFile
Returns a writeStream stream to an output file.
.toFile(output)
Parameters:
output
: the name of the output file.
Alias: toFile
####.toString
Stringifies an Object.
Alias: toString
####.map
Applies a map function to the data. The function must return
something.
.map(data, index)
Parameters:
data
: the data,index
: the index of the data.
Alias: map
Example:
.pipe(map(function (data, index) {
data['age'] = parseInt(data['age']);
return data;
}))
//-> {name: 'Alex', age: '23'} => {name: 'Alex', age: 23}
###CSV
A CSV toolkit. You can use it via oddity.csv
;
Commented sample Oddfile.js
:
var odd = require('oddity');
var csv = odd.csv; // optional, avoids repeating odd.csv later
fromFile('examples/colors.csv') // read from the input file
.pipe(csv.splitLines()) // split the file in lines
.pipe(csv.toArray()) // split each line in arrays
.pipe(csv.zipHeader()) // zip each line with the header
.pipe(toString()) // stringify the JSON
.pipe(toFile('output.json')) // write to the output file
####.splitLines Splits the lines of a CSV file.
.splitLines([options])
Options:
separator
: a String that separates data on each line (defaults to,
)
Example:
.pipe(csv.splitLines({separator: '\r\n'}))
//-> "foo\r\nbar\r\nbaz" => 'foo' -> 'bar' -> 'baz'
####.toArray Splits a string into an array.
.toArray([options])
Options:
start
: the index of the first data to work on (defaults to0
(the first one)),end
: the index of the last data to work on (defaults tonull
(the last one)),separator
: a String that separates data on each line (defaults to,
),quote
: a String that escapes data that contains a separator (defaults to"
).
Example:
//=> pipe(csv.toArray())
//-> "foo,bar,baz" => ['foo', 'bar', 'baz']
####.zipHeader Merges a data Array with a header Array. By default, the header is the first Array that goes through the stream.
.zipHeader([options])
Options:
start
: the index of the first data to work on (defaults to0
(the first one)),end
: the index of the last data to work on (defaults tonull
(the last one)),headerIndex
: the index of the header Array (defaults to0
(the first Array)); if the header is located among the data, the data is buffered to be correctly zipped later,header
: an Array of Strings that defines the header (defaults to the first Array or the Array atheaderIndex
).
Example:
pipe(csv.zipHeader())
//-> ['foo', 'bar', 'baz'] -> ['medical', 'campaign', 'awaiting'] -> ['reasoning', 'second', 'remarked']
//=> {foo: 'medical', bar: 'campaign', baz: 'awaiting'} -> {foo: 'reasoning', bar: 'second', baz: 'remarked'}
####.rotateData Rotates an array of arrays.
Example:
Before rotation:
----------------
// ['iso', 'name'] -> ['FI', 'Finland'] -> ['FR', 'France'] -> ['MA', 'Morocco'] -> ['ES', 'Spain']
//=> .pipe(rotateData())
//=> .pipe(zipHeader())
// {iso: 'FI', name: 'Finland'} -> {iso: 'FR', name: 'France'} -> {iso: 'MA', name: 'Morocco'} -> {iso: 'ES', name: 'Spain'}
After rotation:
---------------
// ['iso', 'FI', 'FR','MA', 'ES'] -> ['name', 'Finland', 'France', 'Morocco', 'Spain']
//=> .pipe(rotateData())
//=> .pipe(zipHeader())
// {FI: 'Finland', FR: 'France', MA: 'Morocco', ES: 'Spain'}
##Testing Testing is done using mocha and should. You can run them by typing:
make test