@twentyfourg/cloud-sdk v2.9.3

V2 is live. See the upgrade notes for information about the breaking changes.

24G Cloud SDK

NPM package to help developers interact with the 24G Cloud Platform.

Development

How to generate types?

npm run tsc wil spit out any type updates. These types are generated from jsdoc doc blocks /** */.

Usage

Logs

Logs emitted while running on the 24G Cloud Platform can be viewed in Datadog if they are formatted properly. Use the logger() function when your application starts to configure console.* functions with proper syntax required by Datadog.

sdk.logger()

Configures console.* functions to work with Datadog.

console.error();
console.warn();
console.info(); // aliased to console.log();
console.http();
console.verbose();
console.debug();
console.silly();

The default log level will log info, warn, and error messages.

const sdk = require('@twentyfourg/cloud-sdk');
sdk.logger();

console.log('OK');
console.error(new Error('error'));

The logger can be setup to use a label in the message.

const sdk = require('@twentyfourg/cloud-sdk');
const logger = sdk.logger('FEATURE-NAME');

logger.info('Processing Complete');

// 23-04-04 13:07:04 [FEATURE-NAME] info: Processing Complete
// {"kind":"application","label":"FEATURE-NAME","level":"info","message":"Processing Complete","timestamp":"23-04-04 13:06:41"}

The logger can be used within the request object through req.log.* and a label can be set with req.logger('FEATURE-NAME').

const express = require('express');
const { middleware } = require('@twentyfourg/cloud-sdk').express;

const app = express();

app.use(middleware());

app.get('/', (req, res) => {
  req.logger('FEATURE-NAME');
  req.log.info('Processing Data');
  res.send();
});

Secrets

Application credentials should be stored outside the code either in Vault or AWS Secret Manager. The Cloud SDK can be used to pull secrets and set them as environment variables.

sdk.secrets(key, options)

Retrieve secrets from Vault or Secret Manager while also setting them as environment variables.

// Single secret
const secrets = await sdk.secrets('kv/test');
console.log(process.env.FOO);

// Multiple secrets
const secrets = await sdk.secrets('kv/test,kv/test2,kv/mysql');
console.log(process.env.FOO);

• key: The key of the secret you are trying to Retrieve. If omitted, the SECRET_PATH environment variable is used.

Storage

Applications running on the 24G Cloud Platform will sometimes need access to object storage for storing assets. The sdk.storage namespace of the SDK will help you interact with object storage for your project.

sdk.storage.put(body, key, options):

Put object in storage.

await sdk.storage.put('hello', 'test.txt', { bucket: 'mybucket' });

• body: String or buffer to write to storage.
• key: The desired name for the object.

sdk.storage.signed.put(key, options)

Creates a signed URL for anonymous uploads.

const url = await sdk.storage.signed.put('test.txt', { bucket: 'mybucket' });

• key: The key of the object you are trying to sign for.

sdk.storage.signed.post(key, options)

Creates a signed URL for anonymous uploads.

const { url, fields } = await sdk.storage.signed.post('assets/image.jpg', {
  conditions: {
    size: { max: 0.5 * 1024 * 1024 },
    starts: [{ key: '$Content-Type', value: 'image/' }],
  },
});

• key: The key of the object you are trying to sign for.
• conditions.equal and conditions.starts values must be in the form of [{ key: '$keyName', value: keyValue }].

sdk.storage.get(key, options)

Retrieve an object from storage.

await sdk.storage.get('test.txt', { bucket: 'mybucket' });
const contents = Body.toString();
/**
{
  LastModified: 2021-07-09T21:49:14.000Z,
  ETag: '"098f6bcd4621d373cade4e832627b4f6"',
  ContentType: 'application/octet-stream',
  ContentLength: 10422,
  Metadata: {},
  Body: <Buffer 74 65 73 74>
}
**/

• key: The object to get.

sdk.storage.signed.get(key, options)

Creates a signed URL for retrieval of private objects from storage.

await sdk.secrets('/kv/test');
const url = await sdk.storage.signed.get('images/profile.png');

• key: The key of the object you are trying to sign for.

sdk.storage.signed.cookie(res, options)

Creates a signed cookie for retrieval of private objects from storage.

sdk.storage.signed.cookie(res);

sdk.storage.list(options)

List objects in storage.

let response = await sdk.storage.list();
/**
{
  IsTruncated: false,
  Contents: [
    {
      Key: 'reputation-management-uploads/',
      LastModified: 2021-03-23T15:06:52.000Z,
      ETag: '"d41d8cd98f00b204e9800998ecf8427e"',
      Size: 0,
      StorageClass: 'STANDARD'
    },
  ],
  next: [Function (anonymous)]
}

**/

If the list of objects is larger than 1000 (or maxKeys), the results will be paginated. The IsTruncated property of the response will be true and you can use the next() method of the response to retrieve the next page.

let allObjects;
let response;

response = await sdk.storage.list({
  bucket: 'cloud-sdk-test-infrastructure-assets',
  maxKeys: 1,
});
allObjects = response.Contents;

while (response.IsTruncated) {
  response = await response.next();
  allObjects = [...allObjects, ...response.Contents];
}
console.log('OBJECTS', allObjects);

sdk.storage.exists(key, options)

Checks if a object with a given key exists in storage. Returns true or false depending on if the object exists

let exists = await sdk.storage.exists('uploads/345a3893fj.jpg');
if(exists) do something

• key: The key of the object to get.

sdk.storage.delete(key(s), options)

Delete object(s) or folder(s) from storage.

// Delete object(s)
await sdk.storage.delete('folder/key');
await sdk.storage.delete(['folder/key1', 'folder/key2']);

// Delete folder(s)
await sdk.storage.delete('folder/');
await sdk.storage.delete('/'); // Deletes everything from the bucket
await sdk.storage.delete.object(['folder1/', 'folder2/']);

• key(s): Single object/folder or a list of objects/folders to delete.

sdk.storage.delete.cookie(res, options)

Deletes signed cookies.

sdk.storage.delete.cookie(res);

Express

sdk.express.start(app, options)

Start a HTTP server given a ExpressJS application.

sdk.express.start(app);

• app: ExpressJS application

sdk.express.middleware()

The top-level middleware function is a wrapper around smaller middlewares, 3 of which are enabled by default.

In other words, these two things are equivalent:

const express = require('express');
const { middleware } = require('@twentyfourg/cloud-sdk').express;

const app = express();

// This...
app.use(middleware());

// ...is equivalent to this:
app.use(middleware.respond());
app.use(middleware.nocache());
app.use(middleware.helmet());
app.use(middleware.cors());
app.use(middleware.accessLogs());

sdk.express.middleware.respond()

Middleware that extends functionality of the native res.send() function.

• Supports String, Object, and Error
• Validates input meets response standards
• Determines status code based off input

res.send({ users: [{ name: 'david' }, { name: 'brian' }] });
// HTTP/1.1 200 OK
{
  "users": [
      {
          "name": "david"
      },
      {
          "name": "brian"
      }
  ]
}

res.send();
// HTTP/1.1 204 No Content

res.send(new Error('this is a custom error'));
// HTTP/1.1 500 Internal Server Error
{
    "error": "this is a custom error",
    "stack": "Error: this is a custom error\n at /cloud-sdk/test.js:14:12"
    "logs": "https://app.datadoghq.com/logs?query=${errorCode}"
}

res.status(404).send(new Error('this is a custom status code'));
// HTTP/1.1 404 Not Found
{
    "error": "this is a custom status code",
    "stack": "Error: this is a custom status code\n at /cloud-sdk/test.js:14:12"
    "logs": "https://app.datadoghq.com/logs?query=${errorCode}"
}

res.send(Error)

When running in prod or qa, res.send(Error) will always result in the error message being "error". This is to prevent the API from leaking production data. All other environments will use error.message.

sdk.express.middleware.errors

Middleware that catches errors and responds using res.send().

const express = require('express');
const { middleware } = require('@twentyfourg/cloud-sdk').express;

const app = express();
middleware.asyncErrors();
app.use(middleware());

// ...api routes

app.use(middleware.errors);

module.exports = app;

sdk.express.middleware.asyncErrors

Middleware that enables express-async-errors.

const express = require('express');
const { middleware } = require('@twentyfourg/cloud-sdk').express;

const app = express();
middleware.asyncErrors();
app.use(middleware());

// ...api routes

app.use(middleware.errors);

module.exports = app;

sdk.express.middleware.nocache()

Middleware that enables nocache.

sdk.express.middleware.helmet(options)

Middleware that enables helmet.

sdk.express.middleware.cors(options)

Middleware that enables cors.

sdk.express.middleware.accessLogs(options)

Middleware that enables Apache styled access logs.

sdk.cache(options)

The Caching namespace provides caching capabilities to your applications backed by Redis or DynamoDB. Both the redis and dynamo caching adapters expose the same functionality and behave the same way, allowing you to easily change your cache storage without having to change any of your code.

The default cache storage is dynamo. You can switch between the caching storage adapters either by using the type option or by using the redis and dynamo sub-namespace.

NOTE: DynamoDB item size limit is 400KB. The SDK will compress the data using zlib if the size exceeds this limit.

Initialize the caching client before you use it.

// Uses Redis: Default storage backend is Redis
const cache = sdk.cache();
// Uses Redis: Redis subnamespace used
const cache = sdk.cache.redis();
// Uses Dynamo: Dynamo subnamespace used
const cache = sdk.cache.dynamo();
// Uses Dynamo: Type option was specified
const cache = sdk.cache({ type: 'dynamo' });

sdk.cache.set(key, data options)

Puts data in a cache. Returns the key used to access the data.

const cache = sdk.cache();
const key = await cache.set('rawr', { foo: 'bar' });

• key: A string key
• data: Data you wish to store at the key.

sdk.cache.get(key)

Retrieve data from cache given a key.

const cache = sdk.cache();
const data = await cache.get('rawr');

• key: A string key

sdk.cache.delete(key)

Removes key/values from cache.

const cache = sdk.cache();
await cache.delete('rawr');

• key: A string key.

sdk.cache.deleteByPattern(pattern)

Removes all key/values from cache that match the pattern.

const cache = sdk.cache();
await cache.deleteByPattern('sessions/*');

• pattern: A pattern to use for searching for keys to delete.

sdk.cache.flush()

Removes all keys from cache

const cache = sdk.cache();
await cache.flush();

sdk.cache.close()

Close connection to the cache

const cache = sdk.cache();
await cache.close();

sdk.cache.middleware(keys)

Returns a ExpressJS middleware that can be added to routes to cache responses.

const cache = sdk.cache();

app.get('/standard', cache.middleware(), (req, res) => {
  console.log(req.cache.hit);
  // Can use any of options from the sdk.cache.set function
  // req.cache.set({ foo: 'bar' }, { ttl: 10 });
  req.cache.set({ foo: 'bar' });
  res.json({ foo: 'bar' });
});

app.get('/custom/:userId', cache.middleware(['headers.authorization']), (req, res) => {
  console.log(req.cache.hit);
  req.cache.set({ foo: 'bar' });
  res.json({ foo: 'bar' });
});

• keys: An array of request properties to append to the cache key. Defaults to the req.originalURL

The caching middleware appends the cache namespace to the Express request object. This namespaces contains the following functions/properties.

• req.cache.hit: Boolean indicating whether it was a cache hit or miss.
• req.cache.set(data,[options]): Function for adding data to the cache for this specific route. Supports that same options as the sdk.cache.set() function.

The X-Cache header is appended to the response object.

< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Cache: HIT

Queues

The queue namespace of the SDK contains functions for interacting with message queues.

sdk.queue(url, options)

Creates a new queue object.

const pointsQueue = sdk.queue('https://pointurl');
const userQueue = sdk.queue('https://userurl.fifo');

• url: The URL of the message queue

sdk.queue.size()

Returns the approximate number of messages on the queue

const numOfMessages = await sdk.queue.size();

sdk.queue.put(message(s))

Put messages on the queue.

When using FIFO queues, deduplication is automatically handled to remove duplicate message ensuring only one copy of the message is delivered. without a groupID each message gets its own group allowing for parallel processing but order is not guaranteed. If you specify a groupID, all messages will use that group and be processed in order.

await sdk.queue.put('singleString');
await sdk.queue.put({ single: 'object' });
await sdk.queue.put(['array', { of: 'things' }]);

await sdk.queue.put('message', { messageAttributes: { containsPassword: true } });

await sdk.queue.put([1, 2, 3, 4, 5], { groupID: 'sessions' });

• message: single string, object or array of strings and objects.

sdk.queue.callback(callback, message(s), options)

Put messages on EZQ which will POST back the messages to the specified callback URL.

await sdk.queue.callback('https://api.com/callback', [{ id: 1 }, { id: 2 }]);

• callback: URL where EZQ will POST message.
• message: single string, object or array of strings and objects.

sdk.queue.query(messages, params, options)

Put SQL messages on EZQ which will insert them into the database.

await sdk.queue.query('INSERT INTO tracking (action, value) VALUES (?, ?)', ['click', 'video']);

await sdk.queue.query([
  ['INSERT INTO tracking (action, value) VALUES (?, ?)', ['view', 'users page']],
  ['UPDATE user SET lastLogin = ? WHERE id = ?', ['2022-03-02 11:47:59', 1]],
]);

await sdk.queue.query('INSERT INTO tracking (action, value) VALUES ?', [
  [
    ['view', 'users page'],
    ['click', 'video'],
  ],
]);

• message: single string, array of strings, or array of arrays.
• params: array of values to escape

sdk.queue.purge()

Delete all messages from the queue.

await sdk.queue.purge();

sdk.queue.get(handler, options)

Retrieve messages from the queue and process them using the provided handler.

await queue.get(async (message) => {
  // Do something with the message.
});

• handler: Async function that called for each message received. | Options | Description | Default | | ----------------- | ------------------------------------------------------------ | ------- | | poll | Whether to consistently poll for message or not. | true | | MaxNumberOfMessages| Max number of message pulled in each batch| 1| | sqs.receiveMessage params| Any valid sqs.receiveMessage params| {}|

The handler function must be async. It must take at least on argument for the message but can optionally have a second for the message's attributes. The message if considered successfully processed if the handler finishes without throwing any errors. Messages are automatically deleted after successful processing.

async function handler(message, [messageAttributes]) {
  // Process the message. Message will be deleted.
  console.log(message);

  // Or...
  // Throw an error if there was an issue
  throw new Error('Error while processing');
}

Slack

The Slack namespace of the SDK contains functions for interacting with the 24G Developer Slack API.

sdk.slack.post(arguments, key)

Sends a message to a channel.

await sdk.slack.post({ channel: 'data-processing', text: 'ETL Job Complete' });

• arguments: Object of Slack arguments used in chat.postMessage method.
• key: The 24G Developer API key. If omitted, the DEVELOPER_API_KEY environment variable is used.

SMS

This namespace of the SDK contains functions for interacting with SMS Text Messages on AWS infrastructure

sdk.sms(applicationID, options)

Creates a new SMS instance.

const sms = sdk.sms('2d16e35ab1e543b68474d0334a258b25');

• applicationID: The Application/Project ID of your PinPoint project

sdk.sms.send(number(s), options)

Sends a SMS text message to a given number or numbers.

Returns an object containing success and failed numbers.

const sms = sdk.sms('2d16e35ab1e543b68474d0334a258b25');
await sms.send('+14254147755', 'Hi there!');

const response = await sms.send(['+14254147755', '+14254147167'], 'Hi there!');
/**
{
  failure: {
    '+12488778282': {
      status: 'PERMANENT_FAILURE',
      messageId: 'ktinqk95s4grb212bpo22vual2rmlo29bkc85jg0',
      statusCode: 400,
      applicationId: 'ec11df4923384a2ebad6da7fc62b58fb',
      requestId: '2e4050df-8922-460a-9bd0-672cc0be0a64'
    }
  },
  success: {
    '+12488778283': {
      status: 'SUCCESSFUL',
      messageId: 't0vm8i7rad22l3si63uo8sm7kmt6fri7acb8o200',
      statusCode: 200,
      applicationId: 'ec11df4923384a2ebad6da7fc62b58fb',
      requestId: '2e4050df-8922-460a-9bd0-672cc0be0a64'
    }
  }
}
 **/

• number(s): Single phone or array of phone numbers

sdk.sms.optIn(number)

Opt back in a number that has been previously opted out. An error will be thrown if the phone number can't be opted out. NOTE: Opted Out Phone numbers can be opted in only once in 30 days

const sms = sdk.sms('2d16e35ab1e543b68474d0334a258b25');
try {
  await sms.optIn('+14254147755');
} catch (error) {
  // Phone number can't be opted back in
}

sdk.sms.eventHandler(handlers, options)

Creates ExpressJS formatted controller that can be use to handle SMS webhook events. The function takes an object of handlers for each type of event.

The controller handles authentication, message decoding and deduplication of events.

If there is an error inside your handler, throw it and the SDK will catch the error, log it, and respond to the webhook with a 500.

const sms = sdk.sms('2d16e35ab1e543b68474d0334a258b25');
app.post(
  '/',
  sms.eventHandler({
    success: async (event) => {
      console.log('IN SUCCESS', event);
    },
    failure: async (event) => {
      console.log('IN FAILURE', event);
    },
    optOut: async (event) => {
      console.log('IN optOut', event);
    },
    buffered: async (event) => {
      console.log('IN buffered', event);
    },
  })
);

• handlers: Object containing functions to be called on each type of event
• [handlers].success: Async function to be called on SUCCESS events
• [handlers].failure: Async function to be called on FAILURE events
• [handlers].optOut: Async function to be called on OPTOUT events
• [handlers].buffered: Async function to be called on BUFFERED events

Note About Origination Numbers and Event Streams

Event streams are set at the PinPoint project/application level. Events, including opting out, is done at the Origination Number level. At the time of writing this, there is no way to explicitly link PinPoint projects and Origination Numbers which can lead to scenarios where SMS events will not reach the desired PinPoint event stream.

Although not officially documented, testing has shown that the last PinPoint project to use an Origination Number will receive it's events. Because of this, it is recommended that each PinPoint project use it's own Origination Number so all events from a single Origination Number will go to the a single PinPoint project's event stream.

Analytics

The analytics namespace provides basic application analytical functionality. The phrase "who did what where" is the main concept behind functionality.

who: 'uid-1234abc`
did: 'button-click`
what: 'register'
where: '/sessions/elearning'
---
who: 'uid-1234abc`
did: 'route-hit`
what: null
where: '/sessions'

A table called tracking should be in your database with the columns who, did, what, and where.

You can create an "instance" of analytics which allows you utilize multiple instances of analytics each with different configuration, or use the function directly off of the namespace.

sdk.analytics(options)

Creates a instances of the analytics namespace.

const sdk = require('@twentyfourg/cloud-sdk');

// this
const analytics = sdk.analytics(options);
analytics.track();
analytics.middleware();
analytics.requestTracking;
analytics.routeTracking();
analytics.endpoint;

// is the same as this but the analytics instance used defaults
sdk.analytics.track();
sdk.analytics.middleware();
sdk.analytics.requestTracking;
sdk.analytics.routeTracking();
sdk.analytics.endpoint;

sdk.analytics.track(event)

Generates a EZQ MySQL query based on a tracking event.

sdk.analytics.track({
  who: 'uid-1234abcd',
  did: 'button-click',
  what: 'signout',
  where: '/sessions',
});

// INSERT INTO tracking (who, did, what, `where`) VALUES ('uid-1234abcd', 'button-click', 'signout', '/sessions');

• [event.who]: "Who" is doing the action. If left blank, the action is consider anonymous.
• [event.did]: The action being done.
• [event.what]: To "What" is the action being done to.
• [event.where]: "Where" did the action happen.

If did, what, and where are all blank, no event is generated.

sdk.analytics.endpoint(req, res, next)

Express middleware that can be mounted to a path allowing frontend application to create tracking events.

app.use(bodyParser.json());
app.post('/tracking', sdk.analytics.endpoint);

// Frontend
axios.post('api.com/tracking, {who, did, what, where})

sdk.analytics.requestTracking(req, res, next)

Express middleware function that appends the analytics namespace to the req object with the track function. The track function is an alias to sdk.analytics.track but gets the who from the request context.

app.use(sdk.analytics.requestTracking);

app.get('/sessions', (req, res, next) => {
  req.analytics.track({ did: 'interested', what: 'more-sessions' });
});

sdk.analytics.routeTracking(options)

Returns a Express middleware function that creates tracking event on every route hit.

app.use(sdk.analytics.routeTracking());

// hit to /sessions route
// {who: 'uid-1234', did: 'route-hit', where: '/sessions'}

sdk.analytics.middleware(options)

Alias function for sdk.analytics.requestTracking and sdk.analytics.routeTracking

// This
app.use(sdk.analytics.middleware());

// Is the same as this
app.use(sdk.analytics.requestTracking);
app.use(sdk.analytics.routeTracking());

WebHooks

The WebHook namespace contains ExpressJS middleware to help handle Web Hooks from various vendors.

AWS

The AWS WebHook middleware can be used to easily handle AWS SNS HTTP events. The body of the request sent to the Web Hook endpoint will vary depending on the AWS source services (S3, CloudWatch, etc.).

sdk.webhook.aws(options)

Returns ExpressJS middleware to handle AWS SNS WebHook requests

app.post('/', sdk.webhook.aws(), (req, res) => {
  // req.body == body of the WebHook
  res.send('OK');
});

AWS Web Hooks requires the Web Hook endpoint to confirm that it would like to receive events. For more information on how to trigger the confirmation process, see these docs.

Event bodies will vary depending on the originating AWS service. For example bodies, see these docs

Environment Variables

Contributing

Contributors are always welcome! If you wish to make an improvement or fix a bug, make a pull request into master. Be sure to follow the commit name convention as you work.

• Slack Channel: #cloud-sdk