@rayvin-flow/flow-scanner v1.0.2

flow-scanner is a standalone service that can monitor the Flow blockchain for one or more Cadence event types and broadcast those events to one or more consumers. You can run it as a standalone service using the CLI, a Docker container, or use the underlying flow-scanner-lib to integrate it into your own application.

Quickstart

You can install the preconfigured standalone CLI version of flow-scanner from NPM:

npm install -g @rayvin-flow/flow-scanner

Check out the Environment Variables documentation below for configuration settings. Once your environment has been configured, you can start the scanner:

flow-scanner

Alternatively, a Docker image is also provided:

The Docker Hub repository can be found at rayvinflow/flow-scanner. The current stable version uses the 1.x tag.

Launch docker image with some default configuration values (check the Environment Variables section for more info):

docker run -e FLOW_ACCESS_NODE=https://access-mainnet-beta.onflow.org -e CADENCE_EVENT_TYPES=A.c1e4f4f4c4257510.TopShotMarketV3.MomentListed -e APP_LOG_LEVEL=debug -e APP_LOG_TYPE=pretty rayvinflow/flow-scanner:1.x

You can also use an .env file (or any other method of setting environment variables for the docker container):

docker run --env-file=.env rayvinflow/flow-scanner:1.x

If you want to build your own docker image, you can follow the instructions below:

• Clone this repository
• Copy the .env.example file to .env and modify the appropriate entries (additional info on this below)
• Run npm install to install all dependencies
• Build the docker image using docker-compose build
• Launch the Docker container using docker-compose up (There is an equivalent command in the Makefile make up, or you can use make up-d to start and detach from the docker container)
• By default, the docker container is watching for changes to the code and will relaunch if any files are modified (using nodemon). If you do not want this, you can launch the container using docker-compose -f docker-compose.yml -f docker-compose.prod.yml up (or make up-prod) to launch in production mode

flow-scanner is easily customized, so check out the Development section below for more information.

Environment Variables

All configuration values are read from the environment (using dotenv). This will look for a .env file in the root of the project, and if you are deploying to AWS (or another cloud provider) then you can set environment variables however is applicable. All available environment variables are dsecribed below, but here is a quick example to configure Top Shot marketplace listings to be broadcast to an HTTP endpoint:

FLOW_ACCESS_NODE="https://access-mainnet-beta.onflow.org"
CADENCE_EVENT_TYPES="A.c1e4f4f4c4257510.TopShotMarketV3.MomentListed"
HTTP_EVENT_BROADCASTER_ENDPOINT="http://localhost/your-http-endpoint"

Required

Optional

Aws Configuration

If you will be using AWS services (SNS, SQS, CloudWatch), then you will need to provide access credentials.

Database Connection

If you are going to be using a database connection (for the settings provider or unique checker), you must define the connection configuration. Knex is used for connecting to the database, and the mysql2 driver is the default. You can use any database that Knex supports, but you will need to modify the application in order to add the necessary driver and update the configuration.

Settings Provider

The settings provider will be used to persist the block height that has been processed by the scanner. If you restart the scanner, it will resume at the last processed block height. You can also use the memory settings service, which will store the currently processed block height in-memory and it will not be persisted across application executions.

Settings Provider (sqlite)

Settings Provider (db)

If you are using the db settings provider, you need to specify the table name to store settings data. The table must already exist, and have the following schema (this example uses a table name of "settings"):

CREATE TABLE `settings` (
  `key` VARCHAR(32) NOT NULL,
  `value` VARCHAR(255) NOT NULL,
  PRIMARY KEY (`key`)
);

Event Broadcaster

An event broadcaster is used to send the events that are scanned from the blockchain to a consumer. You can configure one or more of the event broadcasters described below. If you do not configure an event broadcaster, then all events are logged to the console.

Event Broadcaster (sqs)

If you want to use the AWS SQS event broadcaster, make sure you have configured the AWS credentials described above.

The SQS message will be JSON string in the following format:

type Message = {
  chunkIndex: number // if there are more than 256 events in a transaction, it will be broken into chunks and sent as multiple messages
  chunkCount: number // number of chunks in this transaction
  blockHeight: number // block height of the transaction
  transactionId: string // id of the transaction
  events: [
    blockId: string // id of the block for this event
    blockHeight: number // block height for this event
    blockTimestamp: number // UNIX timestamp for the event
    type: string // Cadence event type
    transactionId: string // id of the transaction
    transactionIndex: number // index of the transaction
    eventIndex: number // index of the event in the transaction
    data: any // decoded payload for the event
  ]
}

Event Broadcaster (sns)

If you want to use the AWS SNS event broadcaster, make sure you have configured the AWS credentials described above.

The SNS message will be JSON string in the following format:

type Message = {
  chunkIndex: number // if there are more than 256 events in a transaction, it will be broken into chunks and sent as multiple messages
  chunkCount: number // number of chunks in this transaction
  blockHeight: number // block height of the transaction
  transactionId: string // id of the transaction
  events: {
    blockId: string // id of the block for this event
    blockHeight: number // block height for this event
    blockTimestamp: number // UNIX timestamp for the event
    type: string // Cadence event type
    transactionId: string // id of the transaction
    transactionIndex: number // index of the transaction
    eventIndex: number // index of the event in the transaction
    data: any // decoded payload for the event
  }[]
}

Event Broadcaster (http)

If you want to broadcast events to an HTTP/s endpoint, define the following environment variables.

The HTTP event broadcaster will send a POST request to the configured endpoint. It will continually attempt delivery until a 200 response is received. The body of the request will be following JSON string:

type Payload = {
  // this is the payload containing the events
  payload: {
    timestamp: number // UNIX timestamp that this payload was sent
    data: {
      transactionId: string // id of the Flow transaction
      blockHeight: number // Flow block height
      events: {
        blockId: string // id of the block for this event
        blockHeight: number // block height for this event
        blockTimestamp: number // UNIX timestamp for the event
        type: string // Cadence event type
        transactionId: string // id of the transaction
        transactionIndex: number // index of the transaction
        eventIndex: number // index of the event in the transaction
        data: any // decoded payload for the event
      }[]
    }
  }
  // this is only included if you have enabled message signing (details below)
  hmac?: {
    nonce: string // nonce to use when verifying the signature
    hash: string // HMAC-SHA512 hash (see below for verification instructions)
  }
}

If your HTTP endpoint is publicly accessible, you may want to enable message verification to make sure you are only processing data that you trust. If you have enabled messaging signing using the HTTP_EVENT_BROADCASTER_SHARED_SECRET environment variable, you can verify the payload using the crypto-js library (or any equivalent library).

import sha256 from 'crypto-js/sha256'
import Base64 from 'crypto-js/enc-base64'
import hmacSHA512 from 'crypto-js/hmac-sha512'

// this should be the body of the request sent to your endpoint
const payload = ...
// this should match the shared secret you defined in the HTTP_EVENT_BROADCASTER_SHARED_SECRET environment variable
const sharedSecret = ...

const message = JSON.stringify(payload.payload)
const hashDigest = sha256(payload.hmac.nonce + message)
const hmacDigest = Base64.stringify(hmacSHA512(message + hashDigest, sharedSecret))

if (payload.hmac.hash === hmacDigest) {
  // the message has been verified
}

HTTP Event Broadcaster Message Deduplication

The SNS and SQS event broadcasters use message deduplication to prevent the delivery of the same events more than once. If you are using HTTP, you can use one of the following providers to perform message deduplication before messages are sent.

Message Deduplication (sqlite)

You can use a SQLite database to eliminate the need for an external database server. The required database/tables will be created automatically at runtime.

Message Deduplication (db)

If you are using the db message deduplication, you need to specify the table name to store deduplication data. The table must already exist, and have the following schema (this example uses a table name of "deduplication"):

CREATE TABLE `deduplication` (
  `key` VARCHAR(128) NOT NULL,
  `lock_id` VARCHAR(64) NULL,
  `lock_timestamp` INT UNSIGNED NULL,
  `consumed` VARCHAR(255) NOT NULL,
  PRIMARY KEY (`key`)
);

CloudWatch Metrics

The scanner can optionally submit metrics for collection and reporting. Integration with AWS CloudWatch is included out-of-the-box.

The following metrics are reported:

Development

flow-scanner is designed to be easily customized to fit into any solution. It is a wrapper around flow-scanner-lib, so check out the docs in that repository for detailed info about the implementation and how to customize the features. A high-level overview of the flow-scanner repository is included below.

Directory structure

./config/env-config.ts This file is responsible for reading all environment variables and putting them into a structured format

./helpers - Helper utility functions

./providers - Providers for the configuration and logging features

./main.ts - The main application code (explained in more detail below)

./server.ts - The entry point for the application

./bin/cli.js - Entry point for the CLI (when installed through npm install)

Customization

The default implementation of flow-scanner is mostly just building a configuration based on the environment variables and passing them to the FlowScanner from flow-scanner-lib. You can review main.ts to check out that implementation. Review the docs for flow-scanner-lib to read more about extending and customizing the scanner.