1.0.2 • Published 4 years ago
batch-stream-csv v1.0.2
batch-stream-csv
Process large CSV files in batches without backpressure
The CSV is streamed in batches and the stream is paused while you perform async operations on each batch to prevent backpressure building up
Uses csv-parser
under the hood
Install
$ npm install batch-stream-csv
Usage:
const batch = require('batch-stream-csv');
batch('./file.csv', batchHandler, options).then(() => console.log('All done!'))
async function batchHandler (batch, progress) => {
console.log(batch) // batch of rows parsed as JSON, e.g. [{ a: 1, b: 2 }, { a: 2, b: 3 }]
await db.batchInsert('table', batch) // the stream is paused while you do your inserts
console.log(`${progress * 100}% complete`) // progress value can be fed into a progress bar
}
API:
Options:
batchSize
: number of rows to process per batch (default = 5000)- ...see https://github.com/mafintosh/ for other options