npm.io
5.0.0 • Published 6 years ago

kafka-streams

Licence
MIT
Version
5.0.0
Deps
8
Size
115 kB
Vulns
1
Weekly
0
Stars
834
SummaryDependencyVersions

node-kafka-streams

Build Status npm version

// suggested Node.js version: v12.16.1
npm install --save kafka-streams
const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(/* .. */);

kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");

CHANGES: The latest version brings a lot of changes, please check here before updating.

API Overview

You might also like

README Overview

Prerequisites

  • Kafka broker should be version >= 0.11.x
  • Node.js should be version >= 8.x.x
A note on native mode

If you are using the native mode (config: { noptions: {} }). You will have to manually install node-rdkafka alongside kafka-streams. (This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)

On Mac OS High Sierra / Mojave: CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4

Otherwise: yarn add --frozen-lockfile node-rdkafka@2.7.4

(Please also note: Doing this with npm does not work, it will remove your deps, npm i -g yarn)

Aim of this Library

  • this is not a 1:1 port of the official JAVA kafka-streams
  • the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
  • stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

Description

kafka-streams equivalent for nodejs build on super fast observables using most.js

ships with sinek for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API

the lib is based on sinek, which is based on kafka-node's ConsumerGroups

Port Progress Overview

  • core structure
  • KStream base - stream as a changelog
  • KTable base - stream as a database
  • KStream & KTable cloning
  • complex stream join structure
  • advanced joins see
  • windows (for joins) see
  • flink like window operations
  • word-count example
  • more examples
  • local-storage for etl actions
  • local-storage factory (one per action)
  • KStorage example for any DB that supports atomic actions
  • backing-up local-storage via kafka
  • kafka client implementation
  • KTable replay to Kafka (produce)
  • stream for topic message production only
  • sinek implementation
  • backpressure mode for KafkaClient
  • auto-json payloads (read-map/write-map)
  • auto producer partition and keyed-message handling
  • documentation
  • API description
  • higher join & combine examples
  • embed native client librdkafka for more performance
  • SSL
  • SASL
  • Kerberos

Operator Implementations

  • map
  • asyncMap
  • constant
  • scan
  • timestamp
  • tap
  • filter
  • skipRepeats
  • skipRepeatsWith
  • slice
  • take
  • skip
  • takeWhile
  • skipWhile
  • until
  • since
  • reduce
  • chainReduce
  • forEach (observe)
  • chainForEach
  • drain
  • _zip
  • _merge
  • _join
  • _combine
  • _sample
  • throttle
  • debounce
  • delay
  • multicast
  • A description of the operators can be found here
  • Missing an operator? Feel free to open an issue

Additional Operators

  • mapStringToArray
  • mapArrayToKV
  • mapStringToKV
  • mapParse
  • mapStringify
  • atThroughput
  • mapWrapKafkaPayload
  • mapToFormat
  • mapFromFormat
  • Want more? Feel free to open an issue

Stream Action Implementations

  • countByKey
  • sumByKey
  • min
  • max
  • Want more? Feel free to open an issue

Join Operations

Operation description

KStream Status
  • merge
  • outerJoin
  • innerJoin
  • leftJoin
  • branch
KTable Status
  • merge
  • outerJoin
  • innerJoin
  • leftJoin
KTable <-> KStream Status
  • merge
  • outerJoin
  • innerJoin
  • leftJoin

Window Operations

KStream
  • window
  • advanced window
  • rolling window

More

Can I use this library yet?

Yes.

Are we ready for production yet?

Probably, yes.

Even More

Forks or Stars give motivation

Keywords

kafka-streamskafkastreamsstreamingtopicsproduceconsumemergejoinmapflatfilterreduceflinkwindowcombine