# db-streamer

> A library to stream data into a SQL database.

Latest version **1.2.1** (published 2018-01-15) · MIT license · 0 weekly downloads

## Install

```sh
npm install db-streamer
pnpm add db-streamer
yarn add db-streamer
bun add db-streamer
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.2.1 |
| Published | 2018-01-15 |
| First published | 2015-09-23 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 3 |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 11 |
| Author | Evan Siroky |
| Maintainers | evansiroky |
| Keywords | database, db, stream, postgresql |

## Links

- npm: https://www.npmjs.com/package/db-streamer
- Repository: https://github.com/evansiroky/db-streamer
- Homepage: https://github.com/evansiroky/db-streamer#readme
- Issues: https://github.com/evansiroky/db-streamer/issues
- npm.io page: https://npm.io/package/db-streamer

## Dependencies (3)

- [async](https://npm.io/package/async.md) ^2.1.2
- [moment](https://npm.io/package/moment.md) ^2.11.2
- [url-parse](https://npm.io/package/url-parse.md) ^1.0.2

## Alternatives

- [angular-pipes](https://npm.io/package/angular-pipes.md) — 5.6K weekly downloads
- [@ng-web-apis/midi](https://npm.io/package/@ng-web-apis/midi.md) — 2.6K weekly downloads
- [happn-3](https://npm.io/package/happn-3.md) — 1.6K weekly downloads
- [@opensip-cli/lang-go](https://npm.io/package/@opensip-cli/lang-go.md) — 1.2K weekly downloads
- [mongoose-typescript](https://npm.io/package/mongoose-typescript.md) — 85 weekly downloads

## Recent versions

- 1.2.1 (latest) — 2018-01-15
- 1.2.0 — 2018-01-14
- 1.1.0 — 2016-11-21
- 1.0.0 — 2016-11-01
- 0.4.5 — 2016-06-21
- 0.4.4 — 2016-06-08
- 0.4.3 — 2016-06-07
- 0.4.2 — 2016-02-29
- 0.4.1 — 2016-02-27
- 0.4.0 — 2015-10-10
- 0.3.3 — 2015-09-25
- 0.3.2 — 2015-09-25
- 0.3.1 — 2015-09-24
- 0.3.0 — 2015-09-24
- 0.2.0 — 2015-09-24
- … 1 more at https://npm.io/package/db-streamer/versions

## README

# db-streamer

[![npm version](https://badge.fury.io/js/db-streamer.svg)](http://badge.fury.io/js/db-streamer) [![Build Status](https://travis-ci.org/evansiroky/db-streamer.svg?branch=master)](https://travis-ci.org/evansiroky/db-streamer) [![Dependency Status](https://david-dm.org/evansiroky/db-streamer.svg)](https://david-dm.org/evansiroky/db-streamer) [![Test Coverage](https://codeclimate.com/github/evansiroky/db-streamer/badges/coverage.svg)](https://codeclimate.com/github/evansiroky/db-streamer/coverage)

A cross-db library to stream data into and out of a SQL database.  Currently supports streaming data into or out of PostgreSQL, MySQL or SQLite.

## Table of Contents

* [Installation](#installation)
* [Inserting](#inserting)
* [Querying](#querying)

## Installation

In order to use this library, you must also install the additional libraries in your project depending on the database that you use.

### PostgreSQL

    npm install pg --save
    npm install pg-copy-streams --save
    npm install pg-query-stream --save
    npm install pg-hstore --save

#### With pg and node v0.10.x

You must also install the package `promise-polyfill` and write additional code.  See [here](https://github.com/brianc/node-postgres/issues/1057) for more details.

### MySQL

    npm install mysql --save
    npm install streamsql --save

### SQLite

    npm install sqlite3 --save
    npm install streamsql --save

#### Deferred inserting w/ SQLite

For now, deferred inserting with SQLite assumes that a unix shell is available to pipe commands to the sqlite3 binary tool.

## Inserting

```js
var dbStreamer = require('db-streamer'),
  connString = 'postgres://streamer:streamer@localhost:5432/streamer-test';

// create inserter
var inserter = dbStreamer.getInserter({
  dbConnString: connString,
  tableName: 'test_table',
  columns: ['a', 'b', 'c']
});

// establish connection
inserter.connect(function(err, client) {

  // push some rows
  inserter.push({a: 1, b: 'one', c: new Date() });
  inserter.push({a: 2, b: 'two', c: new Date() });
  inserter.push({a: 3, b: 'three', c: new Date() });

  // create child table inserter using deferring strategy
  // this is useful to avoid missing foreign key conflicts as a result of race conditions
  var childInserter = dbStreamer.getInserter({
    dbConnString: connString,
    tableName: 'child_table',
    columns: ['a', 'd', 'e'],
    deferUntilEnd: true
  });

  childInserter.push({a: 2, d: 'asdf', e: new Date() });
  childInserter.push({a: 3, d: 'ghjk', e: new Date() });

  childInserter.setEndHandler(callback);

  // set end callback
  inserter.setEndHandler(function() {
    childInserter.end();
  });

  // announce end
  inserter.end();

});
```

### Inserter Config

| Key | Description |
| --- | --- |
| dbConnString | A database connection string. |
| tableName | The tablename to insert into. |
| columns | Array of column names. |
| primaryKey | Required if using MySQL or SQLite.  String of the primary key (defaults to `id` if omitted). |
| deferUntilEnd | Boolean (default=false).  Stream output to temporary file which is then streamed in all at once into table upon calling `end`. |
| sqliteStorage | Required if using SQLite.  String of the filename to load data to.  Unfortunately, will not work with `:memory:` (well it will, but all data will be lost after disconnecting, so it's kind of pointless). |

### Inserter Config (Sequelize Bulk Insert alternative)

| Key | Description |
| --- | --- |
| useSequelizeBulkInsert | Boolean.  Perform the insert using a combination of [async.cargo](https://github.com/caolan/async#cargo) and [sequelize bulkInsert](http://docs.sequelizejs.com/en/latest/api/model/#bulkcreaterecords-options-promisearrayinstance).  Must provide `sequelizeModel` parameter too. |
| sequelizeModel | The sequelize model to perform a bulk insert with. |
| deferUntilEnd | Boolean (default=false).  Pause all cargo iterations until calling `end`. |

## Querying

```js
const querier = dbStreamer.getQuerier({
  dbConnString: 'postgres://streamer:streamer@localhost:5432/streamer-test'
})

querier.execute(
  'SELECT * FROM test_table',
  row => console.log,
  err => {
    console.log('done')
  }
)
```

### Querying Config

| Key | Description |
| --- | --- |
| dbConnString | A database connection string. |
| sqliteStorage | Required if using SQLite.  String of the filename to load data from.  Unfortunately, will not work with `:memory:` (well it will, but a new connection is opened, so there won't be any data to query, so it's kind of pointless). |

---
_Source: https://npm.io/package/db-streamer · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._