@xeedware/aws-lambda-stage NPM

@xeedware/aws-lambda-stage

Utility for preparing node.js AWS Lambda functions and supporting node_module packages for upload to AWS S3.

Overview

Motivation

During our initial attempt at developing a "serverless" webapp backend using AWS Lambda, APIGateway and other AWS services, we could not find a tool to stage and prune our node.js AWS Lambda functions (and supporting files and npm packages) to directories that would then be referenced by a CloudFormation SAM (Serverless Application Model) template for upload to Amazon S3. Thus the birth of aws-lambda-stage.

Why use lambda-stage?\ For obvious reasons, it is in your best interest to keep your Lambda package as small as possible.

When installing production dependencies specified in package.json, the resulting node_modules directory will typically contain extraneous directories and files. For example, try including "env2" and/or "jwt-decode" npm packages in your package.json dependencies then execute npm '--production'. Notice test directories and unnecessary files in the resulting _node_modules subdirectories.

As per your defined Packaging Specs, lambda-stage will copy your Lambda function and supporting files to a staging directory. In addition, production dependent node_module packages defined in package.json will be "installed" in the directory as well but pruned of extraneous directories and files as specified in the Packaging_Specs.

Initial staging runs allows review of resulting content: identifying extraneous directories and files that can be excluded using packaging specs (e.g., excludes and excludePatterns). So iterate through\ 1) reviewing the staging directory content,\ 2) tweaking your Packaging_Specs, and\ 3) running lambda-stage\ until all extraneous directories and files are eliminated from the staged node_modules subdirectories.

For serverless computing, once you've finalized staged directory, you'll typically create a CloudFormation SAM template for use with the AWS CLI aws cloudformation package command that will zip and upload the staged directory to S3. See CloudFormation Package Reference.\ In the SAM (Serverless Architecture Mode) template, the value for the CodeUri property for our AWS::Serverless::Function resource will be the directory path of the staged Lambda package.

Alternatively, you can use the lambda-zip command to create a zip file of your Lambda package and upload the zip file to S3 manually or with a custom script. If you still prefer to use a SAM file, specify the CodeUri value as the path to the zip file. The CloudFormation '--package' command will bypass zipping, directly upload the file to S3.

The Command Line Utilities

This aws-lambda-stage package provides two command line utilities:

lambda-stage\ Stages your Node.js AWS Lambda function, supporting files and production dependent node_modules packages into a directory for subsequent uploading to AWS S3 via the CloudFormation CLI 'package' command. As part of staging, extraneous directories and/or files within node_module packages are pruned as per Packaging Specs. Packaging specifications can handle multiple Lambda functions, resulting in a directory per Lambda function with its supporing node_module packages.
lambda-zip\ Zips the staged directories as per Zip Specs for manual (or custom scripted) upload to AWS S3. If the staging directories specified in the Packaging Specs do not exist, staging will be performed before zipping.

Install

$ npm install @xeedware/aws-lambda-stage --save-dev

The Commands

Once this npm package is installed, two commands are available in node_modules/.bin:

lambda-stage\ Stages your Lambda package into a specified directory.
lambda-zip\ Zips the content of the stage directory into a specified zip file.

These commands reads Staging and Zip Specs specified in an aws-lamba-stage property in your package.json or in a separate file specified with the -specs-file command line option.

Command Line Options

-specs-file _filename*\ Instead of reading the default package.json, use the file specified by filename.\ Useful to "declutter", your package.json file.\ Example:
```
lambda-stage -specs-file myPkgSpecs.json
```
-pkgs name[,name]\ Stage and/or Zip only these specified lambda packages.\ The argument is a single string or comma-delimited strings, each assumed to be a name of a lambda package.\ Example:
```
lambda-stage -pkgs myLambdaPkg1,myLambdaPkg2
```

Usage

You typical use these commands in package.json scripts.\ Examples:

"scripts": {
    "stage": "lambda-stage"
    "zip": "lambda-zip"
}

"scripts": {
    "stage": "lambda-stage -specs-file myPkgSpecs.json"
    "zip": "lambda-zip -specs-file myPkgSpecs.json"
}

Lambda Packaging Specs

Lambda Packaging Specs are defined in the aws-lambda-stage property in your package.json file or in a separate JSON file. The value associated with the aws-lambda-stage property is a JSON object whose base level properties are globals and Lambda package names.

globals: object [optional]\ Value is an object containing the staging and zip properties as described in the next section.
lambdaPkgs: object [required]\ One or more properties whose keys are Lambda package names and value an object containing the staging and zip properties as described in the next section..

globals Staging and Zip Specs

version: string [optional]\ Version number to include in a Lambda Package's zip file name if the Lambda Package's Zip Spec zipFile.version property value is "inherit".\ The semantic version delimiters will be replaced with '-' when generating the file name. For example "1.0.1" will result in "1-0-1".
phase: string [optional]\ String (e.g., "dev", "test", "prod") to be included in a Lambda package's phase portion of the zip file name if the Lambda Package's Zip Spec zipFile.phase property value is "inherit".
buildDir: string [optional]\ Absolute or relative path of the directory where files/directories to be staged are located.\ Defaults to the project root.\ May be overridden by the Lambda Package's Staging Spec buildDir property.
stageDir: string [optional]\ Absolute or relative path of the parent directory for the resulting staging directory. The basename of the staging directory is the Lambda package's name.\ If the staging directory exists, its content will be deleted, clearing it before depositing the new staged artifacts.\ If stageDir or the staging directories do not exist, they will be created.\ stageDir defaults to "lambda-stage" in the project root.\ May be overridden by the Lambda Package's Staging Spec stageDir property.
zipDir: string [optional]\ Absolute or relative path of the directory where the resulting zip file will be placed.\ If the directory does not exist, it will be created.\ Defaults to "lambda-zip" in the project root.\ May be overridden by the Lambda Package's Zip Specs zipDir property.
logLevel: string [optional]\ Specifies the level of logging verbosity: debug, info, warning, error.\ May be overridden by the Lambda Package's Staging and Zip Spec logLevel property.
manifest: object [optional]
includes: Array\<string> [optional]\ Paths specifying files or directories to be included in the stage directory. Paths are relative to buildDir.
excludes: Array\<string> [optional]\ Paths specifying files or directories in the buildDir directory to exclude from the stage directory.
excludePatterns: Array\<string> [optional]\ Each string specifies a Javascript Regex pattern for names of directories or files to be excluded.

lambdaPkgs Staging and Zip Specs

buildDir: string [optional]\ Absolute or relative path of the directory where files/directories to be staged are located.
If specified, overrides the buildDir specified in globals for this Lambda package.\ If not specified here or in globals, buildDir defaults to the project root.
stageDir: string [optional]\ Absolute or relative path of the parent directory for the resulting staging directory.
If specified, overrides the stageDir specified in globals for this Lambda package.\ If not specified here or in globals, stagedir defaults to "lambda-stage" in the project root.
The basename of the staging directory is the Lambda package name.\ If the staging directory exists, its content will be deleted, clearing it before depositing the new staged artifacts.\ If stageDir or the staging directories do not exist, they will be created.
manifest: object [required]
- includes: Array\<string> [required]\ Paths specifying files or directories to be included in the stage directory. These are additions to those specified in globals.manifest.includes property, if any. Paths are relative to buildDir.
- excludes: Array\<string> [optional]\ Paths specifying files or directories in the buildDir directory to exclude from the stage directory. These are additional excludes to those specified in globals.manifest.excludes property, if any.
- excludePatterns: Array\<string> [optional]\ Each string specifies a Javascript Regex pattern for names of directories or files to be excluded. These are additional excludePatterns to those specified in globals.manifest.excludePatterns property, if any.
nodeModules: object [required]\ If undefined, assumes there are no production dependencies, therefore there will be no node_modules directory staged.
Otherwise, lambda-stage copies your package.json to the stage directory then executes npm install --production that will create a node_modules subdirectory containing production dependencies. The package.json is then removed from the stage directory.
Some imported modules may contain extraneous files and directory when installing with the "--production" flag. For example, the "env2" and "jwt-decode" modules contain a test directory and other files not used in the production environment. To reduce the size of the Lambda package, use the following "excludes" and "excludePatterns" properties.
- excludes: Array\<string> [optional]\ Each string specifies the relative path to the directory or file to exclude. These are additional excludes to those specified in globals.nodeModules.excludes property, if any.
- excludePatterns: Array\<string> [optional]\ Each string specifies a Javascript Regex pattern for names of directories or files to be excluded. These are additional excludePatterns to those specified in globals.nodeModules.excludePatterns property, if any.
zipDir: string [optional]\ Absolute or relative path of the directory where the resulting zip file will be placed.
If specified, overrides globals.zipDir if specified.\ If not specified here or in globals, zipDir defaults to "lambda-zip" in the project root.
If the directory does not exist, it will be created.
zipFile: object [optional]\ Specifies the name of the resulting zip file.\ Resulting name will be basename[_version][_phase].zip
- basename: string [optional]\ Specifies the base name of the zip file.\ Defaults to the Lambda package name.
- version: string [optional]\ Version number to include in a Lambda Package's zip file name.\ If undefined here and undefined in the globals section, version number is excluded from the zip file name.\ If undefined here but defined in the globals section, uses value from globals.\ Otherwise the specified string will be included in the zip file name.\ The semantic version delimiters will be replaced with '-' when generating the file name. For example "1.0.1" will result in "1-0-1".
- phase: string [optional]\ String (e.g., "dev", "test", "prod") to be included in a Lambda package's phase portion of the zip file name.\ If undefined here and undefined in the globals section, phase is excluded from the zip file name.\ If undefined here but defined in the globals section, uses value from globals.\ Otherwise the specified string will be included in the file name.
logLevel: string [optional]\ Specifies the level of logging verbosity for this particular package: debug, info, warning, error.

Examples

Single Lambda Package

Example 1a: Minimal Staging & Zip Specs in package.json:

Scripts stage and zip calls the lambda-stage and lambda-zip commands respectively.

The project directory contains the file myProgram.js, therefore executing npm run stage on the command line will result in myProgram.js copied to the ./lambda-stage/myLambdaPkg directory.

Executing npm run zip results in the zip file lambda-zip/myLambdaPkg.zip.

# package.json
{
    .
    .
    .
    "scripts": {
        .
        .
        .
        "stage": "lambda-stage",
        "zip": "lambda-zip",
    },
    .
    .
    .
    "aws-lambda-stage": {
        "lambdaPkgs": {
            "myLambdaPkg": {
                "manifest": {
                    "includes": ["myProgram.js"]
                }
            }
        }
    },
    .
    .
    .
}

Example 1b: Loaded Staging & Zip Specs in package.json:

As in example 1, we add the stage and zip scripts.

Executing npm run stage results in the stageDir myPkgStage/myLambdaPkg containing the file myProgram.js copied from myPkgBuild/myProgram.js and the node_modules directory containing imported production npm packages, less those files and subdirectories specified by the nodeModules.excludes and nodeModules.excludePatterns properties.

Executing npm run zip results in the stages files and subdirectories zipped to the file myPkgZip/MyCustomNamedLambdaPkgZip_0-1-2_dev.zip and with verbose debug appropriate output to the console.

# package.json
{
    .
    .
    .
    "scripts": {
        .
        .
        .
        "stage": "lambda-stage",
        "zip": "lambda-zip",
    },
    .
    .
    .
    "aws-lambda-stage": {
        "globals": {
            "version": "1.0.1",
            "phase": "prod",
            "buildDir": "myMainBuild",
            "stageDir": "myMainStage",
            "stageDir": "myMainZip",
        },
        "lambdaPkgs": {
            "myLambdaPkg": {
                "buildDir": "myPkgBuild",
                "stageDir": "myPkgStage",
                "manifest": {
                    "includes": ["myProgram.js", "lib"],
                    "excludes": ["lib/myFile.js", "lib/myOtherFile.js"],
                    "excludePatterns: [".+\\.ts", ".+\\.xsl", "\\.npmignore", "test"],
                },
                "nodeModules": {
                    "excludes": ["env2/LICENSE", "env2/test"],
                    "excludePatterns": [
                        ".+\\.markdown",
                        ".+\\.md",
                        "\\.travis\\.yml",
                        "AUTHORS",
                        "CONTRIBUTORS",
                        "example",
                        "examples",
                        "LICENSE",
                        "LICENCE",
                        "LICENSE.*\\.txt",
                        "README.mdown",
                        "test",
                        "tests"
                    ]
                },
                "zipDir": "myPkgZip",
                "zipFile": {
                    "basename": "MyCustomNamedLambdaPkgZip",
                    "version": "inherit",
                    "phase": "dev"
                },
                "logLevel": "debug"
            }
        }
    },
    .
    .
    .
}

Multiple Lambda Packages

In many cases, you will be creating multiple Lambda packages in the same project using the same production npm package dependencies.

Currently, this lambda-stage_ utility is intended for use with Lambda functions with similar production dependencies. Differences in uses/unused imported npm packages can be easily handled by excludes and excludePatterns__ specifications.

As with single Lambda packages, you can specify all Lambda packages in package.json or a separate file. But with multiple Lambda packages, you can also specify Staging & Zip Specs for each Lambda package in their own file.

Example 2a: In package.json Only

Scripts and lambda staging & zip specs are in the package.json file.

In the Packaging Specs ("aws-lambda-stage" property) use a unique name for each Lambda package.\ For example "myLambda1", "myLambda2" and "myLambda3".
Add entries to the scripts section in your package.json for each Lambda package/\ Notice the myLambda1-stage and myLambda2-zip scripts that use the -pkg command line option to single out the lambda package on which to operate.

#package.json
{
    .
    .
    .
    "scripts": {
        .
        .
        .
        "stage": "aws-lambda-stage",
        "zip": "aws-lambda-zip",
        "myLambda1-stage": "aws-lambda-stage -pkg myLambda1",
        "mylambda1-zip": "aws-lambda-stage -pkg myLambda1"
    },
    .
    .
    .
    "aws-lambda-stage": {
        "globals": {
        },
        "myLambda1": {
            "manifest": {
                "includes": [ "myLambda1.js", "lib"]
            },
            "zipFile": {
                "version": "inherit",
                "phase": "inherit"
            }
        },
        "myLambda2": {
            "manifest": {
                "includes": [ "myLambda2.js", "lib"]
            },
            "zipFile": {
                "version": "1.0.1",
                "phase": "inherit"
            }
        },
        "myLambda3": {
            "manifest": {
                "includes": [ "myLambda3.js", "lib"]
            },
            "zipFile": {
                "version": "inherit",
                "phase": "test"
            }
        }
    }
}

Example 2b: package.json and single specs file

You can migrate all your staging & zip specs to a separate JSON file and use the -specs-file option in your scripts.

#package.json
{
   .
   .
   .
   "scripts": {
       .
       .
       .
       "stage": "aws-lambda-stage -specs-file lambda.json",
       "zip": "aws-lambda-zip -specs-file lambda.json",
       "myLambda1-stage": "aws-lambda-stage -specs-file lambda.json -pkg myLambda1",
       "mylambda1-zip": "aws-lambda-stage -specs-file lambda.json -pkg myLambda1"
   }
   .
   .
   .
}

and the lambda.json file contains:

#lambda.json
{
    "aws-lambda-stage": {
        "globals": {
            "version": "2.3.4",
            "phase": "dev"
        },
        "lambdaPkgs": {
            "myLambda1": {
                "manifest": {
                    "includes": [ "myLambda1.js", "lib"]
                },
                "zipFile": {
                    "version": "inherit",
                    "phase": "inherit"
                }
            },
            "myLambda2": {
                "manifest": {
                    "includes": [ "myLambda2.js", "lib"]
                },
                "zipFile": {
                    "version": "1.0.1",
                    "phase": "inherit"
                }
            },
            "myLambda3": {
                "manifest": {
                    "includes": [ "myLambda3.js", "lib"]
                },
                "zipFile": {
                    "version": "inherit",
                    "phase": "test"
                }
            }
        }
    }
}

Example 2c: package.json and multiple spec files

In this example, each Lambda package have Packaging Specs in their own file and the package.json have scripts for each Lambda package that use the -specs-file option with the package's associated Specs File.

#package.json
{
   .
   .
   .
   "scripts": {
       .
       .
       .
       "myLambda1-stage": "aws-lambda-stage -specs-file myLambda1.json",
       "mylambda1-zip": "aws-lambda-stage -specs-file myLambda1.json"
       "myLambda2-stage": "aws-lambda-stage -specs-file myLambda2.json",
       "mylambda2-zip": "aws-lambda-stage -specs-file myLambda2.json"
       "myLambda3-stage": "aws-lambda-stage -specs-file myLambda3.json",
       "mylambda3-zip": "aws-lambda-stage -specs-file myLambda3.json"
       "stage": "myLambda1-stage && myLambda2-stage && myLambda3-stage",
       "zip": "myLambda1-zip && myLambda2-zip && myLambda3-zip",
   }
   .
   .
   .
}

and the following spec files:

#myLambda1.json
{
    "aws-lambda-stage": {
        "lambdaPkgs": {
            "myLambda1": {
                "manifest": {
                "includes": [ "myLambda1.js", "lib"]
            },
            "zipFile": {
                "version": "2.3.4",
                "phase": "dev"
            }
        }
    }
}

#myLambda2.json
{
    "aws-lambda-stage": {
        "myLambda2": {
            "manifest": {
                "includes": [ "myLambda2.js", "lib"]
            },
            "zipFile": {
                "version": "1.0.1",
                "phase": "dev"
            }
        }
    }
}

#myLambda3.json
{
    "aws-lambda-stage": {
        "myLambda3": {
            "manifest": {
                "includes": [ "myLambda3.js", "lib"]
            },
            "zipFile": {
                "version": "2.3.4",
                "phase": "test"
            }
        }
    }
}

Notes

aws-sdk dependency\ The node runtime for AWS Lambda includes the aws-sdk module. Therefore when developing your Lambda function, import the aws-sdk with the --save-dev option! This will reduce the size of the zipped Lambda package considerably!