0.2.1 • Published 1 year ago

callmeback v0.2.1

Weekly downloads
-
License
ISC
Repository
github
Last release
1 year ago

callmeback

Serverless-friendly background task processing library. This library lets you enqueue tasks (represented as HTTP POST requests) to be processed in the background by utilizing cloud-based task queue services. Adapters are available for:

Synopsis

// Import the function and adapter you want to use.
import { callMeBack, GoogleCloudTasksAdapter } from 'callmeback'

// Import the SDK to use with the adapter.
import { CloudTasksClient } from '@google-cloud/tasks'

// Define the configuration.
const config = {
  adapter: new GoogleCloudTasksAdapter({
    client: new CloudTasksClient(),
    queuePath: 'projects/callmeback/locations/us-central1/queues/callmeback',
    url: 'https://.../process-background-task',
  }),
  // Other available adapters:
  // - AmazonSNSAdapter
  // - InProcessAdapter
  // - QStashAdapter
  // - ZeploAdapter
}

// Enqueue a background task.
const { id } = await callMeBack(config, {
  // Arbitrary JSON data to be passed to the background task.
  type: 'send-email',
  orderId: '1',
})

// The task ID is generated by the provider, can be used to e.g. search for logs.
console.log(`Task ID: ${id}`)

// Some time later, a POST request will be sent to the URL with
// the specified JSON data as the request body.

Design

Due to differences in the way each service works, this library makes the following trade-offs in order to make it extremely easy to switch to other providers when needed:

  • The request method is always POST.
  • The request body is always JSON-encoded. Some providers doesn’t allow setting request headers, so your endpoint should be configured to always decode JSON body, even if Content-Type is not application/json.
  • The target URL is fixed. Many providers lets you set the target URL on a per-task basis, but some requires a target URL to be pre-configured (e.g. Amazon SNS). On providers that allows setting the URL for each task, the URL can be configured directly on the adapter (e.g. Google Cloud Tasks).
  • Retry logic depends on the service. Amazon SNS has a default retry policy but it can be configured on a topic or subscription. On Google Cloud Tasks, a RetryConfig can be configured on a queue.
  • Rate limiting (throttling) depends on the service. Amazon SNS lets you configure a throttle policy at the topic or subscription level. Google Cloud Tasks lets you configure RateLimits on a queue.
  • Backpressure. The provider will call your endpoint as fast as it could (unless throttling is enabled). Your service should be configured to send a 504 (Gateway Timeout) response if it is overloaded. 429 (Too Many Requests) is not recommended because some providers will consider all 4xx responses as permanent failures and stop retrying.
  • It is your service’s responsibility to verify the authenticity of incoming requests. An easy way is to embed some secret key when invoking callMeBack() and verify that the secret key is present on the way back. Or use some kind of signature or JWT.
  • Due to retrying, your service may receive duplicate requests. It is your responsibility to make sure that your background job can be properly retried (e.g. by deduplicating requests or making sure actions are idempotent or conflict-free).
  • Your service should respond in 10 seconds. Some providers will consider a request as failed if it takes too long to respond. If your service needs more time to process a request. If it will take more than 10 seconds, you should split the work into multiple tasks.

Usage with Google Cloud Tasks

About Google Cloud Tasks:

Setting up:

  1. Create a task queue.

    image

  2. Give it a name and region.

    image

    Take note of the name, region ID, and project ID (found on the dashboard).

  3. Construct a queue path using this format:

    projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID

  4. Grant access to the queue:

    image

Creating an adapter:

import { CloudTasksClient } from '@google-cloud/tasks'

const adapter = new GoogleCloudTasksAdapter({
  client: new CloudTasksClient(),
  queuePath: process.env.CALLMEBACK_CLOUD_TASK_QUEUE,
  url: 'https://.../',
})

Expected environment variables:

# PROJECT_ID is the ID of the Google Cloud project, found on the Google Cloud Console dashboard.
# LOCATION_ID is the ID of the location where the queue is located, e.g. "us-central1".
# QUEUE_ID is the ID of the queue, e.g. "callmeback".
CALLMEBACK_CLOUD_TASK_QUEUE=projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID

# When providing service account credentials to the SDK via environment variables.
# Note that there may be better ways to provide credentials to the SDK
# depending on your environment and use case.
# See: https://cloud.google.com/docs/authentication/provide-credentials-adc
GOOGLE_APPLICATION_CREDENTIALS=

Common errors:

  • Error: Could not load the default credentials.
  • Error: 3 INVALID_ARGUMENT: Invalid resource field value in the request.
    • May be fixed by correctly configuring the queuePath (i.e. the CALLMEBACK_CLOUD_TASK_QUEUE environment variable) and making sure the url is valid.
  • Error: 7 PERMISSION_DENIED: The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/…/locations/…/queues/…" (or the resource may not exist).
    • May be fixed by making sure the queue exists, the queuePath is correctly configured, permission to create tasks in the queue is granted to the user or service account (by using the “Cloud Tasks Enqueuer” role).

Monitor the metrics in Cloud Tasks dashboard:

image

Check the logs in Google Cloud Logging:

image

Inspecting the outstanding tasks in Cloud Tasks dashboard:

image

Usage with Amazon SNS

About Amazon SNS:

Setting up:

  1. Create a topic:

    image

  2. Set it as a standard queue:

    image

  3. Take note of the topic ARN. Create a subscription.

    image

  4. Make it an HTTPS subscription and set an endpoint. Make sure to enable the “Raw message delivery” option.

    image

  5. Confirm the subscription by checking the SubscribeURL.

    image

  6. Grant permission to access the queue.

    image

Creating an adapter:

import { SNS } from '@aws-sdk/client-sns'

const adapter = new AmazonSNSAdapter({
  topicArn: process.env.CALLMEBACK_SNS_TOPIC_ARN,
  sns: new SNS({}),
})

Expected environment variables:

CALLMEBACK_SNS_TOPIC_ARN=

# When providing credentials to the SDK via environment variables.
# Note that there may be better ways to provide credentials to the SDK
# depending on your environment and use case.
# See: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-iam.html
AWS_REGION=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

Common errors:

  • Error: Region is missing
    • May be fixed by setting the AWS_REGION environment variable.
  • CredentialsProviderError: Could not load credentials from any providers
  • AuthorizationErrorException: User: … is not authorized to perform: SNS:Publish on resource: … because no identity-based policy allows the SNS:Publish action
    • May be fixed by granting the user permission to publish to the topic.
  • InvalidParameterException: Invalid parameter: TopicArn or TargetArn Reason: no value for required parameter
    • May be fixed by correctly configuring the topicArn (i.e. the CALLMEBACK_SNS_TOPIC_ARN environment variable).
  • NotFoundException: Topic does not exist
    • May be fixed by creating the topic (and making sure the topic ARN is correctly configured).

Monitor metrics in CloudWatch:

image

Setting up logging:

image

Usage with QStash

About QStash:

Setting up:

  1. Copy the API key

    image

Creating an adapter:

const adapter = new QStashAdapter({
  url: 'https://.../',
  token: process.env.QSTASH_TOKEN,
  retry: 3,
})

Expected environment variables:

QSTASH_TOKEN=

Inspecting requests in the dashboard:

image

Running tests

By default, the tests will only run against the in-process adapter.

To run the tests against other adapters, there are some preparation steps.

  1. Run the requestbin script: node scripts/requestbin.mjs

    This script starts a web server that lets the tests verify the behavior.

  2. Expose the requestbin service so that it’s publicly available.

    • If you are developing in Codespaces, you can go to the Ports tab, right click on the server’s Local Address → Port Visibility → Public.
    • You can also use ngrok or Cloudflare Quick Tunnel to expose the service.

    Then set the environment variable REQUESTBIN_URL to the URL of the requestbin service without the trailing slash, for example:

    export REQUESTBIN_URL=https://dtinth-verbose-computing-machine-rr4g7rgqv2jgj-35124.preview.app.github.dev

  3. Set up the credentials for the other adapters.

    • Check out the above sections for more details.

Usage with Zeplo

About Zeplo:

  • Very easy to configure.
  • The free plan allows up to 500 requests per month.
  • Flexible retry policies.
  • Provides an easy-to-use dashboard for viewing tasks.
  • Allows inspecting the response body and headers.
  • Provides a CLI with the zeplo dev simulator that allows local development.

Setting up:

  1. Copy the API key

    image

Creating an adapter:

const adapter = new ZeploAdapter({
  zeploUrl: process.env.ZEPLO_URL,
  url: 'https://.../',
  token: process.env.ZEPLO_TOKEN,
  retry: 3,
})

Expected environment variables:

ZEPLO_TOKEN=

# If you want to test locally with `zeplo dev`, uncomment the next line:
ZEPLO_URL=http://localhost:4747

Inspecting requests in the dashboard:

image

@everything-registry/sub-chunk-1288
0.2.1

1 year ago

0.2.0

1 year ago

0.1.0

1 year ago

0.1.0-0

1 year ago