0.8.0 • Published 9 years ago
text-pipe v0.8.0
text-pipe
Transform stream enumeratee generators for stream-driven data extract and transformation (i.e. ETL).
$ npm install text-pipe
text-pipe is a node module that includes functions for creating enumeratees (i.e. map, reduce) that serve as transform streams. This allows you to iteratively compose complex data processing workflows via simple map, reduce, and filter-like functions. Check out the examples folder for working code samples. In order to run the code samples, you must do the following:
% git clone https://github.com/wilmoore/text-pipe.git
% cd text-pipe
% npm install
% cd examples
% cat stackoverflow.txt
% node extract-question-ids
NOTE: read the comments and the code in the examples to get a feel for how it works.
API
.map
aliases: .transform
For each value, return result of calling transformer
function to given value.
var transform = map(function transformer(string) {
return /\d+/.exec(string)[0];
});
stream.pipe(transform);
.select
aliases: .filter, .keep
For each value, return only values that pass a truth test.
var transform = select(function (string) {
return /^question_id:/.exec(string);
});
stream.pipe(transform);
.reduce
aliases: .fold, .inject
Array.prototype.reduce
as a through stream.
var max = reduce(function(memo, val) {
return Number(val) > Number(memo) ? val : memo;
}, 0)
stream.pipe(max);
Depends on
- stream-reduce
- through2-filter: You could use this directly if you want to write
{wantStrings: true}
boilerplate. - through2-map: You could use this directly if you want to write
{wantStrings: true}
boilerplate. - through2: You could use this directly if you want to write
write
andend
boilerplate.