# @mathquis/node-shovel > Shovel messages from here to there Latest version **2.1.4** (published 2023-04-25) · ISC license · 0 weekly downloads ## Install ```sh npm install @mathquis/node-shovel pnpm add @mathquis/node-shovel yarn add @mathquis/node-shovel bun add @mathquis/node-shovel ``` Provides the command `shovel`. ## Health **Score 20/100 (F)** — status: abandoned. Positive: esm support; no vulnerabilities. Warnings: low downloads; no types. Negative: abandoned; low maintenance score. ## Facts | | | |---|---| | Version | 2.1.4 | | Published | 2023-04-25 | | First published | 2021-03-18 | | Weekly downloads | 0 | | License | ISC | | TypeScript types | none | | Module format | ESM + CommonJS | | Dependencies | 39 | | Unpacked size | 142.3 KB | | Known vulnerabilities | 0 (+2 in 2 direct dependencies) | | Install scripts | no | | GitHub stars | 0 | | Author | Mathieu "Fastjack" Quisefit | | Maintainers | mathquis | ## Links - npm: https://www.npmjs.com/package/@mathquis/node-shovel - Repository: https://github.com/mathquis/node-shovel - Homepage: https://github.com/mathquis/node-shovel#readme - Issues: https://github.com/mathquis/node-shovel/issues - npm.io page: https://npm.io/package/@mathquis/node-shovel ## Dependencies (39) - [koa](https://npm.io/package/koa.md) ^2.14.2 - [cron](https://npm.io/package/cron.md) ^2.3.0 - [cuid](https://npm.io/package/cuid.md) ^2.1.8 - [glob](https://npm.io/package/glob.md) ^10.2.2 - [mqtt](https://npm.io/package/mqtt.md) ^4.3.7 - [pupa](https://npm.io/package/pupa.md) ^3.1.0 - [tail](https://npm.io/package/tail.md) ^2.2.6 - [clone](https://npm.io/package/clone.md) ^2.1.2 - [json5](https://npm.io/package/json5.md) ^2.2.3 - [level](https://npm.io/package/level.md) ^8.0.0 - [luxon](https://npm.io/package/luxon.md) ^3.3.0 - [colors](https://npm.io/package/colors.md) ^1.4.0 - [amqplib](https://npm.io/package/amqplib.md) ^0.10.3 - [convict](https://npm.io/package/convict.md) ^6.2.4 - [js-yaml](https://npm.io/package/js-yaml.md) ^4.1.0 - [winston](https://npm.io/package/winston.md) ^3.8.2 - [msgpackr](https://npm.io/package/msgpackr.md) ^1.8.5 - [cli-table](https://npm.io/package/cli-table.md) ^0.3.11 - [csv-parse](https://npm.io/package/csv-parse.md) ^5.3.8 - [node-fetch](https://npm.io/package/node-fetch.md) ^3.3.1 - [protobufjs](https://npm.io/package/protobufjs.md) ^7.2.3 - [@koa/router](https://npm.io/package/@koa/router.md) ^12.0.0 - [prom-client](https://npm.io/package/prom-client.md) ^14.2.0 - [events-async](https://npm.io/package/events-async.md) ^1.2.1 - [memory-level](https://npm.io/package/memory-level.md) ^1.0.0 - [csv-stringify](https://npm.io/package/csv-stringify.md) ^6.3.2 - [json-colorizer](https://npm.io/package/json-colorizer.md) ^2.2.2 - [koa-basic-auth](https://npm.io/package/koa-basic-auth.md) ^4.0.0 - [koa-bodyparser](https://npm.io/package/koa-bodyparser.md) ^4.4.0 - [nsyslog-parser](https://npm.io/package/nsyslog-parser.md) ^0.10.1 - [parse-duration](https://npm.io/package/parse-duration.md) ^1.0.3 - [promise-events](https://npm.io/package/promise-events.md) ^0.2.4 - [elasticsearch-5](https://npm.io/package/elasticsearch-5.md) npm:@elastic/elasticsearch@^5.6.22 - [elasticsearch-6](https://npm.io/package/elasticsearch-6.md) npm:@elastic/elasticsearch@^6.8.8 - [elasticsearch-7](https://npm.io/package/elasticsearch-7.md) npm:@elastic/elasticsearch@^7.17.0 - [elasticsearch-8](https://npm.io/package/elasticsearch-8.md) npm:@elastic/elasticsearch@^8.7.0 - [whatwg-mimetype](https://npm.io/package/whatwg-mimetype.md) ^3.0.0 - [level-read-stream](https://npm.io/package/level-read-stream.md) ^1.1.0 - [@elastic/elasticsearch](https://npm.io/package/@elastic/elasticsearch.md) ^8.7.0 ## Recent versions - 2.1.4 (latest) — 2023-04-25 - 2.1.3 — 2022-07-19 - 2.1.2 — 2022-07-01 - 2.1.1 — 2022-03-24 - 2.1.0 — 2022-03-23 - 2.0.0 — 2022-03-17 - 1.1.7 — 2022-02-10 - 1.1.6 — 2022-02-10 - 1.1.5 — 2021-07-13 - 1.1.4 — 2021-07-12 - 1.1.3 — 2021-07-12 - 1.1.2 — 2021-04-13 - 1.1.1 — 2021-03-23 - 1.1.0 — 2021-03-23 - 1.0.9 — 2021-03-19 - … 9 more at https://npm.io/package/@mathquis/node-shovel/versions ## README # shovel [![Tests](https://github.com/mathquis/node-shovel/actions/workflows/node.js.yml/badge.svg)](https://github.com/mathquis/node-shovel/actions/workflows/node.js.yml) ## Installation `npm i @mathquis/node-shovel` ## Usage `shovel --pipeline pipeline.yml --config config.yml --workers 1 --log-level debug --metrics-port 3001` ### Prometheus metrics Pipeline metrics are exposed as Prometheus format on the specified port ### Pipeline configuration ```yaml name: message workers: 2 input: use: amqp options: vhost: '/' username: rabbitmq password: password exchange_name: exchange exchange_type: topic queue_name: queue bind_pattern: '#' decoder: use: parser.js options: pipeline: use: pipeline.js options: encoder: use: noop options: queue: use: batch options: persistent: true # Store queue on disk batch_size: 1000 flush_timeout: 5s output: use: elasticsearch options: scheme: http index_name: audit-events-{YYYY}-{MM} template: template.js username: elastic password: password ``` Pipeline configuration can use environment variables like so `${NAME:default}`. ### Available inputs - amqp - file - http-request - http-server - mqtt - noop - stdin - stream - syslog - tcp - udp ### Available decoders (optional) - base64 - csv - json - json5 - multiline (WIP) - noop - protobuf ### Available encoders (optional) - base64 - csv - format - json - json5 - noop - protobuf ### Available queues (optional) - batch - noop ### Available outputs - amqp - blackhole - debug - elasticsearch - file - mqtt - pipeline - stdout - tcp - udp ### Node ```javascript export default node => { node // Use convict schema .registerConfig({}) // Create a new message object .createMessage() // Events: start, stop, up, down, pause, resume, in, ou, ack, nack, ignore, reject, error .on(event, handler) .off(event, handler) .once(event, handler) // When the node starts (if set, the handler is responsible for calling node.up()) .onStart(async () => {}) // When the node stops .onStop(async () => {}) // When the node is up (connected, ready, etc.) .onUp(async () => {}) .up() // When the node is down (disconnected, unable to process messages) .onDown(async () => {}) .down() // When the node should pause processing messages .onPause(async () => {}) .pause() // When the node should resume processing messages .onResume(async () => {}) .resume() // When the node receives a message .onIn(async (message) => {}) .in(message) // When the node push a message down the pipeline .onOut(async (message) => {}) .out(message) // When the node acks a message .onAck(async (message) => {}) .ack(message) // When a node nacks a message .onUnack(async (message) => {}) .unack(message) // When the node ignores a message .onIgnore(async (message) => {}) .ignore(message) // When the node rejects a message .onReject(async (message) => {}) .reject(message) // When the node triggers an error .error(err) } ``` ### Decoder ```javascript export default node => { node .registerConfig({}) .onIn(async (message) => { message.decode(decodedValue) node.out(message) }) } ``` ### Encoder ```javascript export default node => { node .registerConfig({}) .onIn(async (message) => { message.encode(encodedValue) node.out(message) }) } ``` ### Pipeline ```javascript export default node => { node .registerConfig({ enabled: { doc: '', format: Boolean, default: true }, blocked: { doc: '', format: Boolean, default: true } }) .onIn(async (message) => { const {blocked} = node.getConfig() if (blocked) { // Reject message node.reject(message) } else if (!node.getConfig('enabled')) { // Ignore message node.ignore(message) } else { // Process message node.out(message) } }) } ``` --- _Source: https://npm.io/package/@mathquis/node-shovel · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._