1.0.0 • Published 9 years ago
lambda-job v1.0.0
LambdaJob
LambdaJob provides real-time invoking and completion detection of AWS Lambda functions from JavaScript on the clientside's web browser. This makes it easy to scale things like: scraping websites, thumbnail generation of images/videos, running custom binaries, etc.
Features
- Lambdas triggered by S3 Object creation
- Real-time results via SQS long-polling
- No back-end needed, just an AWS account and public IAM permissions
- Low-latency and performance focus throughout
- Maximum job throttling (to stay under the 25 Lambda limit)
- Shared SQS queue per client
- Helper for calling commandline applications on Lambda
Usage
From the web browser:
var lambdaJob = new LambdaJobClient();
lambdaJob.invoke("bash", {cmd: "/sbin/ifconfig"}, function (err, output) {
console.log(output);
});
The Lambda job:
var LambdaJob = require('lambda-job');
var lambdaJob = new LambdaJob.LambdaJobWorker(AWS, jobReceived);
exports.handler = lambdaJob.lambdaHandler;
function jobReceived(params, errDataCallback) {
console.log("Will run: bash -c " + params.cmd + "...");
lambdaJob.execHelper(params.cmd, function(err, consoleOutput) {
errDataCallback(err, consoleOutput);
}
}
Outputs in the web browser:
vsb_20 Link encap:Ethernet HWaddr 26:3A:68:14:69:21
inet addr:192.168.20.21 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54 errors:0 dropped:0 overruns:0 frame:0
TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:23982 (23.4 KiB) TX bytes:10948 (10.6 KiB)
[...]
Options
debugLog
: Turn debug logging on/off. Useful to turn on to see how LambdaJob works. (default: false)maxActiveJobs
: Throttle how many jobs can run at once. Note AWS has a hard limit of 25 active jobs at any one point. (default: 25)jobTimeout
: The maximum amount of time (in ms) that a job can run for. (default: 30000)s3BucketPrefix
: Since we create S3 objects for every desired job, the bucket they're inside of has this prefix. Note that S3 buckets are in a global namespace with other people. (default: lambda-job-)sqsQueuePrefix
: Like the s3BucketPrefix except for SQS queues. (default: lambda-job-)
Prerequisites
Create an IAM user with the following permissions:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1418615018000", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:DeleteObject", "sqs:CreateQueue", "sqs:DeleteQueue", "sqs:ReceiveMessage", "sqs:SendMessage", "sqs:DeleteMessage" ], "Resource": [ "arn:aws:s3:::lambda-job-*", "arn:aws:sqs:us-west-2:YOURACCOUNTNUM:lambda-job-*" ] } ] }
Create an S3 bucket named
lambda-job-LAMBDANAME
Add CORS permissions on the S3 bucket. Here's a CORS config sample:
<?xml version="1.0" encoding="UTF-8"?> <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <CORSRule> <AllowedOrigin>*</AllowedOrigin> <AllowedMethod>GET</AllowedMethod> <AllowedMethod>POST</AllowedMethod> <AllowedMethod>PUT</AllowedMethod> <AllowedMethod>DELETE</AllowedMethod> <AllowedMethod>HEAD</AllowedMethod> <MaxAgeSeconds>3000</MaxAgeSeconds> <AllowedHeader>*</AllowedHeader> </CORSRule> </CORSConfiguration>
Create your Lambda and set this S3 bucket as your S3 source. It's recommended to use the full 1024MB of RAM since it seems things are faster this way.
- In your Lambda, upload your files to AWS, making sure you included the npm modules you'll need in the archive.
- Browse to index.html either where you've hosted it, or locally.
Examples
- bash shows how you can run custom binaries on Lambda. It also provides convenient access to explore the Lambda VM and what's available.
- nomocors shows how you could use LambdaJob to proxy around CORS restrictions. Great for client-side automation.
1.0.0
9 years ago