kafka-datagen-test v0.1.1

Datagen CLI

Installation

npm install
npm link

Usage

datagen -h
Usage: datagen [options]

Fake Data Generator

Options:
  -V, --version                output the version number
  -f, --format <char>          The format of the produced data (choices: "json", "avro", default: "json")
  -s, --schema <char>          Schema file to use
  -n, --number <char>          Number of records to generate (default: "10", infinite records: "-1")
  -d, --debug <char>            (choices: "true", "false", default: "false")
  -w, --wait <int>             Wait time in ms between record production (default: "0")
  -dr, --dry-run <char>        Dry run (no data will be produced (choices: "true", "false", default: "false")
  -rs, --record-size <char>    Record size in bytes, eg. 1048576 for 1MB
  -h, --help                   display help for command

Env variables

To produce records to a Kafka topic, you need to set the following environment variables:

SASL_USERNAME=
SASL_PASSWORD=
SASL_MECHANISM=
KAFKA_BROKERS=

Examples

# Generate 10 records in JSON format
datagen -s products.sql -f json -n 10

Output:

✔  Parsing schema...


✔  Creating Kafka topic...


✔  Producing records...


✔  Record sent to Kafka topic
  {"products":{"id":50720,"name":"white","merchant_id":76809,"price":1170,"status":89517,"created_at":"upset"}}
  ...

JSON Schema

The JSON schema option allows you to define the data that is generated using Faker.js.

[
    {
        "_meta": {
            "topic": "mz_datagen_users"
        },
        "id": "datatype.uuid",
        "name": "internet.userName",
        "email": "internet.exampleEmail",
        "phone": "phone.imei",
        "website": "internet.domainName",
        "city": "address.city",
        "company": "company.name",
        "age": "datatype.number",
        "created_at": "datatype.datetime"
    }
]

The schema needs to be an array of objects, as that way we can produce relational data in the future.

Each object represents a record that will be generated. The _meta key is used to define the topic that the record will be sent to.

You can find the documentation for Faker.js here

Record Size Option

In some cases, you might need to generate a large amount of data. In that case, you can use the --record-size option to generate a record of a specific size.

The --record-size 1048576 option will generate a 1MB record. So if you have to generate 1GB of data, you run the command with the following options:

datagen -s ./tests/datasize.json -f json -n 1000 --record-size 1048576

This will add a recordSizePayload key to the record with the specified size and will send the record to Kafka.

Note: The 'Max Message Size' of your Kafka cluster needs to be set to a higher value than 1MB for this to work.

UPSERT Evelope Support

To make sure UPSERT envelope is supported, you need to define an id column in the schema. The value of the id column will be used as the key of the record.

Faker.js and SQL Schema

The SQL schema option allows you to define the data that is generated using Faker.js by defining a COMMENT on the column.

CREATE TABLE "ecommerce"."products" (
  "id" int PRIMARY KEY,
  "name" varchar COMMENT 'internet.userName',
  "merchant_id" int NOT NULL COMMENT 'datatype.number',
  "price" int COMMENT 'datatype.number',
  "status" int COMMENT 'datatype.boolean',
  "created_at" datetime DEFAULT (now())
);

The COMMENT needs to be a valid Faker.js function. You can find the documentation for Faker.js here.

Docker

Build the docker image.

docker buildx build -t datagen .

Run a command.

docker run \
  --rm -it \
  -v ${PWD}/.env:/app/.env \
  -v ${PWD}/tests/schema.json:/app/blah.json \
    datagen -s blah.json -n 1 -dr true