# kafka-connect > kafka-connect is a framework to move data via kafka Latest version **4.0.0** (published 2020-02-24) · MIT license · 0 weekly downloads ## Install ```sh npm install kafka-connect pnpm add kafka-connect yarn add kafka-connect bun add kafka-connect ``` ## Health **Score 25/100 (F)** — status: abandoned. Positive: has types; no vulnerabilities; high quality score. Warnings: low downloads; no esm support. Negative: abandoned; low maintenance score. ## Facts | | | |---|---| | Version | 4.0.0 | | Published | 2020-02-24 | | First published | 2017-05-13 | | Weekly downloads | 0 | | License | MIT | | TypeScript types | bundled | | Module format | CommonJS | | Dependencies | 7 | | Unpacked size | 58.5 KB | | Known vulnerabilities | 0 (+5 in 3 direct dependencies) | | Install scripts | no | | GitHub stars | 127 | | Author | Chris Froehlingsdorf | | Maintainers | krystianity | | Keywords | kafka, connect, framework, move, data, postgres, cassandra, bigquery, stream, fluent | ## Links - npm: https://www.npmjs.com/package/kafka-connect - Repository: https://github.com/nodefluent/kafka-connect - Homepage: https://github.com/nodefluent/kafka-connect#readme - Issues: https://github.com/nodefluent/kafka-connect/issues - npm.io page: https://npm.io/package/kafka-connect ## Dependencies (7) - [async](https://npm.io/package/async.md) ~3.2.0 - [debug](https://npm.io/package/debug.md) ~4.1.1 - [sinek](https://npm.io/package/sinek.md) ~9.0.0 - [express](https://npm.io/package/express.md) ~4.17.1 - [bluebird](https://npm.io/package/bluebird.md) ~3.7.2 - [body-parser](https://npm.io/package/body-parser.md) ~1.19.0 - [prom-client](https://npm.io/package/prom-client.md) ~12.0.0 ## Alternatives - [gamedig](https://npm.io/package/gamedig.md) — 29.3K weekly downloads - [join-monster](https://npm.io/package/join-monster.md) — 12.8K weekly downloads - [masked](https://npm.io/package/masked.md) — 5.5K weekly downloads - [@comunica/actor-query-process-explain-logical](https://npm.io/package/@comunica/actor-query-process-explain-logical.md) — 4.7K weekly downloads - [@veracity/vui](https://npm.io/package/@veracity/vui.md) — 4.6K weekly downloads ## Recent versions - 4.0.0 (latest) — 2020-02-24 - 3.9.0 — 2020-02-23 - 3.8.0 — 2019-05-07 - 3.7.0 — 2019-01-09 - 3.6.0 — 2018-10-16 - 3.5.1 — 2018-09-20 - 3.5.0 — 2018-09-10 - 3.4.0 — 2018-06-27 - 3.3.1 — 2018-03-27 - 3.3.0 — 2018-03-15 - 3.2.0 — 2017-12-04 - 3.1.0 — 2017-10-13 - 3.0.3 — 2017-08-30 - 3.0.2 — 2017-08-30 - 3.0.1 — 2017-08-29 - … 28 more at https://npm.io/package/kafka-connect/versions ## README # node-kafka-connect [![Build Status](https://travis-ci.org/nodefluent/kafka-connect.svg?branch=master)](https://travis-ci.org/nodefluent/kafka-connect) [![Coverage Status](https://coveralls.io/repos/github/nodefluent/kafka-connect/badge.svg?branch=master)](https://coveralls.io/github/nodefluent/kafka-connect?branch=master) ## What can I do with this? The framework can be used to build connectors, that transfer data `to` and `from` Apache Kafka and Databases, very easily. If you are looking for already implemented connectors for you favorite datastore, take a look at the `Available Connector Implementations` below. ## Info - node-kafka-connect is a framework to implement large `kafka -> datastore` & `datastore -> kafka` data movements. - it can be used to easily built connectors from/to kafka to any kind of datastore/database. - a connector might consist of a SourceConnector + SourceTask to poll data from a datastore into a kafka topic. - a connector might consist of a SinkConnector + SinkTask to put data from a kafka topic into a datastore. - Converters might be used to apply alteration to any data-stream. - any operation in node-kafka-connect is asynchronous - ships with auto http server (health-checks, kafka-stats) - ships with auto metrics (prometheus) ## A note on native mode If you are using the native mode (`config: { noptions: {} }`). You will have to manually install `node-rdkafka` alongside kafka-connect. (This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1) On Mac OS High Sierra / Mojave: `CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4` Otherwise: `yarn add --frozen-lockfile node-rdkafka@2.7.4` (Please also note: Doing this with npm does not work, it will remove your deps, `npm i -g yarn`) ## Available Connector Implementations * [Sequelize (MySQL, Postgres, SQLite, MSSQL)](https://github.com/nodefluent/sequelize-kafka-connect) * [Google BigQuery](https://github.com/nodefluent/bigquery-kafka-connect) * [Salesforce](https://github.com/nodefluent/salesforce-kafka-connect) * [Google PubSub](https://github.com/nodefluent/gcloud-pubsub-kafka-connect) ## Creating custom Connectors ``` yarn add kafka-connect ``` ```es6 const source = new TestSourceConfig(config, TestSourceConnector, TestSourceTask, [TestConverter]); source.run().then(); ``` ```es6 const sink = new TestSinkConfig(config, TestSinkConnector, TestSinkTask, [TestConverter]); sink.run().then(); ``` ## Docs * [Implementation-Helper Overview](docs/sample.md) * [Framework Events](docs/events.md) ## Debugging * You can use `DEBUG=kafka-connect:*` to debug the sink configuration. ## FAQ * Q: it is running slow / only synchronous / 1 by 1 messages ? * A: just set the config.batch object [as it is described here](https://github.com/nodefluent/node-sinek/tree/master/lib/librdkafka#advanced-1n-consumer-mode) --- _Source: https://npm.io/package/kafka-connect · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._