faros-feeds-sdk v1.4.5

feeds-sdk

An SDK that simplifies feed development by handling all Faros API interactions. A feed is a tool that extracts data from an external data source, typically an IaaS or PaaS API, and loads it into Faros.

Uploading data

The SDK includes two functions for uploading data to the Faros platform: withEntryUploader and entryUploader. They have slightly different signatures, but they both wrap all the necessary API calls to the Faros platform, so you can focus on what's unique to your feed. Choosing one comes down to whether your feed requires state to be persisted between runs.

Stateless feeds

Feeds which do not need to persist state between runs can use entryUploader. This will give you a stream.Writable whose entries are uploaded to the API.

Example of using entryUploader:

import {entryUploader} from 'faros-feeds-sdk';

const writer = await entryUploader({
  name: 'EC2',
  // Other required parameters...
});

// Written entries will be streamed to the API
writer.write({
  ec2_Instance: {
    instanceId: 'i-0ef827cbfe244614f',
    instanceType: 't2.micro',
    state: {
      code: 80,
      name: 'stopped',
    },
    launchedAt: new Date(1607473249000),
  },
});

// Write more entries and finally:
writer.end();

Stateful feeds

When possible, it's good practice to support incremental data ingestion. In order to do so, use withEntryUploader, which comes with a callback you can use to read the existing feed state, and return the new state.

Example of using withEntryUploader:

import {withEntryUploader} from 'faros-feeds-sdk';

export interface FeedState {
  readonly lastLaunchedAt: number;
}

const uploaderCfg = {
  name: 'EC2',
  // Other required parameters...
};
await withEntryUploader<FeedState>(uploaderCfg, (writer, state) => {
  // Write entries to writer using the old state...
  return /* new state */;
});

Lambda runner

The feeds SDK also includes a convenience function called createLambdaHandler for generating a lambda handler if you would like to run your feed in Faros's Lambda environment.

:clipboard: Note: The resulting lambda handler is only compatible with feeds that have been set up and scheduled via Faros's /accounts API, because of the very specific way in which it handles feed params.

The lambda handler decrypts secrets, params, and environment variables provided by the Faros scheduler, and passes them on to a user defined runFeed method that in turn uses them to run the feed. Example usage:

import {createLambdaHandler, FarosEnv} from 'faros-feeds-sdk';

async function runFeed(env: FarosEnv): Promise<void> {
  // validate input params
  // run the feed
}

export const lambdaHandler = createLambdaHandler(runFeed);

An example FarosEnv for the GitHub feed might look something like this:

{
  "token": "myGitHubToken",
  "orgLogin": "acme",
  "repoList": ["repo1", "repo2"],
  "farosApiUrl": "https://prod.api.faros.ai",
  "farosApiKey": "myFarosApikey",
  "runMode": "Root",
  "logger": Logger
}

Worker Feeds

The createLambdaHandler also takes in an optional parameter partition, which is a function will takes the FarosEnv and returns a list of FarosEnv to enable parallel execution of the feed. The root feed should adapt each sub FarosEnv with the appropriate param the feed needs to run.

:clipboard: Note: Create unique values of the param the feed uses as the origin to ensure correct handling of the worker's feed state.

import {createLambdaHandler, FarosEnv} from 'faros-feeds-sdk';

async function runFeed(env: FarosEnv): Promise<void> {
  // validate input params
  // run the feed
}

async function partition(env: FarosEnv): Promise<Map<string, FarosEnv>> {
  // return map of farosEnv with split params
  // e.g. return new Map([['partition1', farosEnv]])
}

export const lambdaHandler = createLambdaHandler(runFeed, partition);

Logging

The FarosEnv passes a logger to the feed. This logger should be used by the feed for all logging to ensure logs are correctly stored.

Links

1. Canonical models. This library can be used for the sdl property of EntryUploaderConfig. See the docs for details.