@corva/local-testing-framework v4.0.0

Local Testing Framework

Local Testing Framework (LTF) is designed to develop Corva Lambda applications and generate workload on the local environments and remote machines.

Table of Contents

• Local Testing Framework
  • Table of Contents
  • Features
  • Requirements
    • Software
    • Utility Docker Images
  • Usage
    • Install
    • Update
    • Configuring datasources
      • External MongoDB
      • Local MongoDB
        • CSV JSON
        • JS
    • Preparing your Lambda Application
      • Dockerizing
      • Building an Image
      • Container Registry
    • Preparing corva-api-lite
      • Building Docker Image
    • Prepare Local Testing Framework
    • Running your App
      • Config File
      • Running
    • Workflow
    • Exploring the outputs

Features

• Support for the following app types:
  • Real-Time (Stream, based on WITS data)
  • Polling (Scheduled)
• Languages: Javascript (Node.js), Python, .NET, Java, etc. (full list of supported languages: https://aws.amazon.com/lambda/faqs/)

Requirements

Software

• >=node.js-12.0.0
• >=docker-18.0.0
• >=docker-machine-0.13.0 - (optional) if you plan to launch lambdas on external machines (e.g. AWS EC2, DigitalOcean and so on)

Utility Docker Images

• corva-api-lite - simplified and fast Corva API.
• redis - for apps that need to save state between runs

Usage

Install

Use npm to install the application globally:

npm i -g @corva/local-testing-framework

Update

If there are no breaking changes:

npm update -g @corva/local-testing-framework

in other case you need to reinstall the LTF

Configuring data sources

Currently the app supports external MongoDB, local CSV, JSON files, and MongoDB's JS scripts.

External MongoDB

To use external MongoDB, first of all, you should disable local MongoDB setup by setting infra.mongo.enabled to false.

{
    infra: {
        mongo: {
            enabled: false,
        },
    }
}

And in source.type select mongo and provide connection string in its options

{
    source: {
        type: 'mongo', // source for generating events
        options: {
            mongo: {
                url: 'mongodb://external-mongo.com',
                db: 'corva',
            },
        }
    }
}

Local MongoDB

To use local MongoDB enable it's setup by setting infra.mongo.enabled to true.

{
    infra: {
        mongo: {
            enabled: true,
        },
    }
}

Then you should specify the directory of file location with data you want to import.

CSV JSON

For CSV (TSV) and JSON data, you need to specify the directory where the data you want to import is located. It should follow the next structure

import-dir/
├── database_1/
│   ├── collection_1.csv
│   ├── collection_2.json
│   ├── collection_3.tsv
│   └── collection_4.json
└── database_2/
    ├── collection_5.json
    └── collection_6.tsv

NOTE if you're exporting JSON data from MongoDB, use --jsonArray flag to export valid JSON.

Then in settings, choose directory for source.type and provide a path to it, either absolute or relative, to the directory where you launched your app.

{
    source: {
        type: 'directory', // source for generating events
        options: {
            directory: {
                // directories inside = database names
                // files inside directories = collection names
                path: './example/mongo/directory',
            },
        }
    }
}

JS

This app allows importing data into local MongoDB by writing MongoDB scripts. Comparing to CSV/JSON import, it will enable more advanced configuration of your data.

Here's an example of such script:

**
 * @var {import('mongodb').Db} db
 */

var error = true;

/**
 * @type {import('mongodb').Db}
 */
// @ts-ignore
var mydb = db.getSiblingDB('test-database');
/**
 * @type {import('mongodb').Collection}
 */
// @ts-ignore
var myCollection = mydb.testcollection;

var res = [
    myCollection.drop(),
    myCollection.createIndex({ myfield: 1 }, { unique: true }),
    myCollection.createIndex({ thatfield: 1 }),
    myCollection.createIndex({ thatfield: 1 }),
    myCollection.insert({ myfield: 'hello', thatfield: 'testing' }),
    myCollection.insert({ myfield: 'hello2', thatfield: 'testing' }),
    myCollection.insert({ myfield: 'hello3', thatfield: 'testing' }),
    myCollection.insert({ myfield: 'hello3', thatfield: 'testing' }),
];

// @ts-ignore
printjson(res);

if (error) {
    // @ts-ignore
    print('Error, exiting');
    // @ts-ignore
    quit(1);
}

To use this kind of import, you should provide the path to a directory with such JS files.

NOTE: During startup, MongoDB will execute files in alphabetical order. Use prefixes if you want to define some specific order.

{
    source: {
        type: 'manual', // source for generating events
        options: {
            manual: {
                path: './example/mongo/manual',
            },
        },
    }
}

Preparing your Lambda Application

For Node.js based example see project in examples directory.

Dockerizing

To run Lambda App via LTF, you'll need to create a Docker image. Use the following example:

FROM lambci/lambda:nodejs12.x
COPY . /var/task

If you need other languages than Node, you can find them on dockerhub

Building a Docker Image

To build your image, you need to run the following command

 docker build -t <lambda_name> .

Replace <lambda_name> with your actual Lambda App name, so it will be easier to recognize what images do you have.

If you would like to build an image on each local-testing-framework run, you need to add lambda.context option into the corvarc file to point to the directory where the sources are.

Container Registry

If you want to run your lambda on 3rd party servers (not locally), you'll need to push it to some external container registry from where your server could download an image.

See https://docs.docker.com/engine/reference/commandline/push/ and https://www.docker.com/blog/how-to-use-your-own-registry/ for more details

Preparing corva-api-lite

In order to avoid significant load on Corva API you should use corva-api-lite server that exposes the same HTTP interface as Corva API. It operates directly with the local MongoDB database.

Building Docker Image for API lite

1. Install local-testing-framework npm i -g @corva/corva-api-lite
2. Build docker image using docker build -t corva-api-lite .
3. (Optionally) Push it to an external registry if you plan to run your app on a remote machine

Prepare Local Testing Framework

1. Install local-testing-framework npm i -g @corva/local-testing-framework
2. (Optionally) View cli app help via node bin/corva.js --help and node bin/corva.js local --help

Running your App

Before running your app you should decide what type of events does it consume. Currently, app re-runner supports Real-Time (Stream) and Polling (Scheduled) apps.

Here are some examples of such events:

Stream Event:

[
  {
    "metadata": {
      "apps": {
        "corva.wits-depth-summary": {
          "app_connection_id": 123
        }
      },
      "app_stream_id": 456
    },
    "records": [
      {
        "asset_id": 1,
        "timestamp": 1546300800,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 99.4,
          "weight_on_bit": 1,
          "state": "Some unnecessary drilling that's excluded"
        }
      },
      {
        "asset_id": 1,
        "timestamp": 1546300800,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 99.4,
          "weight_on_bit": 1,
          "state": "Rotary Drilling"
        }
      },
      {
        "asset_id": 1,
        "timestamp": 1546300900,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 99.5,
          "weight_on_bit": 1,
          "state": "Rotary Drilling"
        }
      },
      {
        "asset_id": 1,
        "timestamp": 1546301000,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 99.9,
          "weight_on_bit": 1,
          "state": "Rotary Drilling"
        }
      },
      {
        "asset_id": 1,
        "timestamp": 1546301100,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 100.3,
          "weight_on_bit": 1,
          "state": "Rotary Drilling"
        }
      },
      {
        "asset_id": 1,
        "timestamp": 1546301200,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 100.5,
          "weight_on_bit": 1,
          "state": "Rotary Drilling"
        }
      },
      {
        "asset_id": 1,
        "timestamp": 1546301300,
        "company_id": 24,
        "version": 1,
        "data": {
          "hole_depth": 100.6,
          "weight_on_bit": 1,
          "state": "Rotary Drilling"
        }
      }
    ]
  }
]

Scheduled Event:

{
  "collection": "operations",
  "source_type": "drilling",
  "environment": "qa",
  "interval": 300,
  "schedule_start": 1578420000000,
  "schedule_end": 1578420300000,
  "asset_id": 16280
}

LTF based on app type generates one of those event types. By default, for streams, it queries WITS collection and creates events from it from the very beginning of the well until drilling is completed. For scheduled events, it gets intervals between first and last wits record and generates 300s events within a given interval.

Also, it's possible to launch the app for a single event. See LTF help for how to.

Config File

To specify most options, you may create .corvarc.js or corvarc.json in your $CWD.

module.exports = {
  lambda: {
    // image: 'app_name', // use this if you already have the prebuilt lambda docker image, can be overriden with CLI argument
    // context: '.', // use this if you wish to build lambda image automatically on each run
    type: 'stream',
    handler: 'lambda_function.lambda_handler', // entrypoint of your lambda
    env: ['A=B'], // command options will override it, not add
  },
  infra: {
    stopContainers: true,
    runner: 'local',
    redis: {
      enabled: true,
      image: 'redis:latest',
      // url: 'redis://127.0.0.1' // for external redis
    },
    apiLite: {
      enabled: true,
      image: 'corva-api-lite:latest',
      source: 'mongo',
    },
    aws: {
      region: 'us-east-1',
      // secretAccessKey: 'xxx',
      // accessKeyId: 'yyy',
    },
  },
  source: {
    type: 'mongo', // source for generating events
    options: {
      mongo: {
        url: 'mongodb://127.0.0.1',
        options: {
          // useNewUrlParser: true,
          // useUnifiedTopology: true,
        },
        db: 'corva',
      },
      limits: {
        // source specific queries
        timestamp: {
          $gt: 0,
        },
      },
    },
    collection: 'corva#wits', // source collection
  },
  debugOutput: 'console', // console | file | none - write debug data to console or file
  event: {
    // json: '', // pass event json here to run single event
    assetId: 1234,
    companyId: 1,
    sourceType: 'drilling',
    appKey: 'app key used in lambda config',
    limits: {
      // specify limits for generated events (e.g. to skip some), format [key]: {...limits}
      timestamp: {
        $gt: 0,
      },
    },
    config: {
      batchSize: 10, // 4 stream events
      interval: 300, // 4 scheduled events
      start_time: 1602765873127, // optional, process events only occuring after this point of time
      end_time: 1602765973127, // optional, process events only occuring before this point of time
      start_depth: 10, // optional, process events only occuring after this hole depth
      end_depth: 17000, // optional, process events only occuring before this hole depth
    },
    options: {
      // additional stuff that could be passed to event
      // useful for scheduled events for example
      timezone: 'America/Chicago',
      collection: 'corva#destination-collection', // 4 scheduled events
    },
  },
  export: {
    enabled: true,
    format: 'json',
    collections: ['corva#destination-collection'],
  },
};

Running

Here's an example how to launch LTF:

node bin/corva.js local --lambda.env MY_ENV_VARIABLE="<some_value>" --lambda.image=test-lambda

Or with LTF locally installed:

corva local --lambda.env MY_ENV_VARIABLE="<some_value>" --lambda.image=test-lambda

1. --lambda.image=test-lambda - the name/uri of docker image of your application
2. --lambda.env MY_ENV_VARIABLE="<some_value>" - environment variables that will be passed to lambda docker container

You may override values from the config file with CLI options.

Useful variables:

• LOG_LEVEL controls on which log messages should be printed (info by default), e.g. LOG_LEVEL=debug will allow debug, info, warn and error messages, while LOG_LEVEL=error will permit error messages only
• LOG_THRESHOLD_MESSAGE_SIZE controls on how many symbols will be printer per logger invocation (1000 by default), e.g. LOG_THRESHOLD_MESSAGE_SIZE=10000. This will not affect the deployed app.
• LOG_THRESHOLD_MESSAGE_COUNT controls on how many log messages will be allowed to print per lambda invocation (15 by default), e.g. LOG_THRESHOLD_MESSAGE_COUNT=100. This will not affect the deployed app.

NOTES:

Priority of the environment variables:

• CLI lambda.env options
• .corvarc file lambda.env options

• settings.env section from manifest.json file.

• the CLI lambda.env options will replace .corvarc provided options

• CLI/corvarc will extend manifest.json provided options
• only settings.env section from manifest.json will be applied on app deployment

Workflow

In general, LTF follows the next steps

1. Setup Docker using Docker machine or use local
2. Download/check for all needed images (redis, mongo, corva-api-lite, your app)
3. Launch Mongo
4. Launch Redis
5. Launch corva-api-lite
6. Prepare env variables for Lambda application
7. Get WITS data and determine bounds for events
8. Start loop
   1. Create an event
   2. Create a container for Lambda App
   3. Pass env variables to the container
   4. Run Lambda app container
   5. Shutdown container
   6. Back to 7.1
9. (OPTIONAL) Export data to json/csv
10. Shutdown all containers and exit

Exploring the outputs

• Lambda run output is piped to the console stdout by default. It is possible to redirect output to file by setting the debugOutput config option to file.
• Infrastructure containers like mongo, redis, etc., will be removed after the test run by default. To preserve containers running, please override the option infra.stopContainers and set it to true.

Exporting run data

It is possible to export the content of the MongoDB collections in JSON or CSV format.

The results of export will be available in output directory in your $CWD, the file format is $LAMBDANAME$ASSETID$TIMESTAMP_$COLLECTION.\$FORMAT, e.g dev-center-gamma-depth_1234_1615369834679_corva#data.drillstring.csv

To export in JSON set export property to

{
  enabled: true,
  format: 'json',
  collections: ['corva#example1', 'corva#example2] // list of collections you want to export
}

To export in CSV set export property to

{
  enabled: true,
  format: 'csv',
  collections: ['corva#example1', 'corva#example2'] // list of collections you want to export
  fields: {
    'corva#example1': ['field1', 'field2', 'deep.structure.field'],
    'corva#example2': ['some-other-field', 'field_with_dashes']
  }
}