omnivore v0.0.2

omnivore

Sample code for using Transloadit to encode any file added to an S3 bucket (using Lambda)

Warning the code in this repo should be thought of more as an inspiration than a ready-to-rock batte-tested spaceship. Use it accordingly, and at own risk!!

The meat to setup all the plumbing is in ./main.sh. This is written in Bash so we can easily turn this into a tutorial, people can pick just the commands they're interested in, and that it will likely be easy to pickup for users coming from all different backend languages.

The actual function that's fired for every S3 upload is in ./omnivore.js. This is written in Node.js since that was the primary Lambda function as it's a language/platform the Transloadit team is most comfortable with. It could easily be adapted to use a different language and Transloadit SDK, but likely: You're never touching this function, if you have different encoding requirements, you would change the Template in your Transloadit account. So basically, you set this up once, make sure it works, and don't look back.

Prerequisites

# developed using versions: awslogs@0.10 awscli@1.14.60 jq@1.5_2
brew install awslogs awscli jq

The aws commands shown in main.sh assume that admin permissions to create/configure: Lambda functions, roles, S3 buckets are set up already via environment variables or ~/.aws/credentials/.

Install

git clone git@github.com:transloadit/omnivore.git
cd omnivore
yarn || npm i

Add the following information to env.sh:

TRANSLOADIT_KEY=x         # Transloadit user account id
TRANSLOADIT_SECRET=x      # Transloadit user account secret
TRANSLOADIT_TEMPLATE_ID=x # Transloadit template to run on the S3 files
OMNIVORE_SRCBUCKET=x      # the bucket name to monitor for newly added files

If you forget to do this, you'll be asked when running the setup.

Deploy Lambda function, role, S3 bucket, notification config

• npm run setup to idempotently (create if not exists, update if exists) create the required Lambda function, role, S3 bucket, notification config. The Lambda function itself takes the S3 bucket name from the notification payload, but to setup the permissions, etc we need to know the source bucket at setup too.
• Note: a destination bucket isn't needed, this is defined in the Transloadit Template (can be changed in ./env.sh, and then redeployed via npm run setup)

Test

• npm run upload to test an upload of ./chameleon.jpg to the source bucket. This should trigger a notification and fire the Lambda function inside ./omnivore.js that we already deployed using setup.
• npm run run:local to run the ./omnivore.js function locally using a ./dummy-payload.json
• npm run run:remote invokes the Lambda function on AWS from your console, using a ./dummy-payload.json (this requires the ./chameleon.jpg to already be present in your source bucket)

Monitor

• npm run logs:audit to check for any permission issues that the Lambda function may have encountered
• npm run logs:tail to monitor logs of the invoked function (best open in this in a new tab, while running uploads or:)

Troubleshooting

The specified log group does not exist

If you run into this while doing npm run logs:tail, it's possible Omnivore didn't run yet and hence did not create any log (groups/streams). Try doing a setup first, and then an upload.

Sources

• https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
• https://docs.aws.amazon.com/AmazonS3/latest/user-guide/setup-event-notification-destination.html
• https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
• https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html
• https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
• https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html