Mgmt-assets NPM

mgmt-assets

This module manages your app's assets that are compiled on a per-commit-basis. This allows you to keep only the core files in Git and have this module compile and hash the derivative assets automatically in your CI. It also keeps track of the location of the asset (local or remote) based on the environment (dev, staging, production) so you can easily change these paths in your server.

Examples of assets include:

Browserify and Webpack builds
Multiple versions of images, ex. favicons
sitemap.xml if you want to compile it every time

Because all your assets are compiled in your CI, some extra benefits include:

You don't need to npm install your frontend dependencies in production.
Save all your frontend dependencies as devDependencies.
Use builds that are known to work in the testing environment.

Both will speed up deployments a tiny bit.

Overview

Publishing Steps

These are the steps that occur when you "publish" your assets.

Run all "compile" steps
Get the current commit SHA
Push all commit-SHA-based assets to S3
Push all SHASUM-based assets to S3
Create a list of all the SHASUM-based assets pushed to S3 and save a JSON metadata object based on the commit-SHA

Runtime Steps

These are the steps that happen when you try to access the location of an asset.

If the asset is local, return the local path. Note that if the asset is not compiled, it will 404. You are expected to compile all the assets yourself during development.
If the asset is Commit-SHA-based

Get the commit-SHA if we haven't already
Return the remote path

If the asset is SHASUM-based

Get the commit-SHA if we haven't already
Get the JSON metadata object for this commit-SHA if we haven't already
Return the remote path

Commit-SHA-based Paths

You can push your assets to S3 based on the commit SHA. This means that every (tested) commit will create a new set of assets.

For example, if your current commit SHA is ff5eac9630c2d2f3f130e38202f2b7df734a8111 and your asset is called index.js, then the asset will be pushed to s3://<bucket>/ff5eac9630c2d2f3f130e38202f2b7df734a8111/index.js.

SHASUM-based Paths

You can push your assets to S3 based on the SHA-256 sum of each file. This is more beneficial for files that don't change often (such as images and logos) and for files that are very big.

For example, you would save your index.js as s3://<bucket>/ff5eac96/index.js.

Costs and Benefits

mgmt-assets will always run all your "compile" steps on publish. Keep this in mind if the operations are expensive.
If your assets are gigantic, use SHASUM-based paths to reduce duplication.
You could separate your commit-SHA-based and your SHASUM-based assets into different buckets, then have your commit-SHA-based buckets expire after X days.
Your CI builds will be slower.

Configuration

Currently, the configuration file must be a .js file called mgmt-assets.js. A .js file is required because you need environment variables which shouldn't be saved in a JSON file.

Environment

The current environment, defaulting to exports.environment = process.env.NODE_ENV || 'development'. This is primarily used for accessing the location of an asset at runtime. You only need to set exports.environment= if you have different environments than those set via process.env.NODE_ENV.

Compile

These are a list of compile steps to be run.

Strings will be executed through a child process.
Functions will be resolved as a Promise.
Objects will be executed sequentially based on Object.keys().
Arrays will be executed in parallel.

// all of these steps will be run in parallel
exports.compile = [
  'NODE_ENV=production webpack --optimize-dedupe --optimize-minimize',
  './bin/create-favicons',
]

Note that in development, you should have your jobs that run these steps. For example, have a npm run watch or webpack --watch step for development.

Buckets

An object of buckets stored by name. Each bucket object is passed directly to knox.

// NOTE: the first bucket you define will be set as the default if haven't set a default
exports.buckets = {
  default: {
    key: '<api-key>',
    secret: '<secret>',
    bucket: '<bucket>',
  }
}

// if you only have one, this is the same as `exports.buckets = { default: {} }`
exports.bucket = {
  key: '<api-key>',
  secret: '<secret>',
  bucket: '<bucket>',
}

Pipelines

Pipelines push assets from specific folders to specific buckets. Each pipeline may differ based on:

Location of the folder
Asset matching
Mount path of the folder on the server
Bucket

Each pipeline has the following options:

name REQUIRED - a name for this pipeline for debugging purposes
root REQUIRED - root folder to look for files in, relative to CWD
files=['*'] OPTIONAL - list/globs of files.
format=':commit/:name' REQUIRED - the format to store assets in the bucket.
- :name - the name of the file, ex. favicon.ico
- :commit - store based on the git commit. For example, :sha/:name will store favicon.ico as something like dab5e497a2bec6815e8648838681820eb2bb8664/favicon.ico
- :branch - store based on the git branch name. For example, :branch/:name will store favicon.ico as something like master/favicon.ico
- :sha(length) - store based on the SHA-256 sum of the file, truncated to length characters. The default length is 8. For example, :sha(10)/:name
- format = filename => string - instead of passing a format string, create your arbitrary formatter by returning a function that is passed the filename and returns a string of the format above.
bucket='default' OPTIONAL - bucket to store assets in. This bucket should be defined in exports.buckets or exports.bucket.
cacheControl='public, max-age=31536000' OPTIONAL - optional cache control headers
storageClass='STANDARD' - the storage class of the files. Use 'REDUCED_REDUNDANCY' if you don't mind a lower durability for the savings cost, which should be fine for this use-case since all your files are derivatives - just publish again!
paths={} REQUIRED - path prefix based on environments. Trailing / is required. The special case is true, which automatically sets the path of the bucket. For example:

{
  paths: {
    default: '/', // image.png -> served as /image.png
    staging: true, // serve from S3
    production: 'https://cdn.example.com/assets/', // image.png -> served as https://cdn.example.com/assets/image.png
  }
}

Example:

exports.pipelines = [
  {
    root: 'build', // all the built files form my `npm build` command
    paths: {
      default: '/',
      staging: true,
      production: 'https://cdn.example.com/',
    }
  }
]

Metadata Bucket

The bucket to store all the builds' metadata, defaulting to the default bucket. All metadata will be stored in the form :commit/assets.json.

exports.metadata_bucket = 'default'

Concurrency

The maximum number of files to upload to S3 at the same time. The higher, the faster, but you will also be more susceptible to rate limits. By default, the value is 5.

exports.concurrency = 5

Commands

CLI

mgmt-assets publish

mgmt-assets publish

Compiles the assets and pushes them to S3.

mgmt-assets compile

mgmt-assets compile

Just compiles the assets.

JS API

const Assets = require('mgmt-assets').assets()

Assets.getPath('index.js').then(asset => {
  return asset.getPath()
})

const Asset = Assets.assets(config, options)

Creates an instance of the getAsset object based on a configuration file.

If no config is passed, it looks for require(path.resolve('mgmt-assets.js')).
options - if a boolean, a shortcut for options.remote. If a string, a shortcut for options.commit.
- remote=config.environment === !~['development', 'test'].indexOf(this.config.environment) - whether to check the remote builds for assets.
- commit=<current commit> - the commit to check assets for.

Asset.getPath(name, pipeline).then(url => {})

Get the path of the asset.

name=<String> REQUIRED - the name of the file
pipeline=<String> OPTIONAL - the pipeline this file is located in. If not defined, will look through all the pipelines.

CI Configuration

npm install --save-dev mgmt-assets
After you run your tests, run ./node_modules/.bin/mgmt-assets push

circle.yml

https://circleci.com/docs/configuration

test:
  post:
    - ./node_modules/.bin/mgmt-assets push

.travis.yml

https://docs.travis-ci.com/user/customizing-the-build/

after_script:
  - ./node_modules/.bin/mgmt-assets push

Tips

Babel Cache

If you're using webpack and babel, you should also setup a caching directory in babel-loader and cache it in your CI system. This will make your frontend builds faster as your builds are cached.

Heroku

Heroku deletes the .git folder, so you won't have access to the current git commit. To get it, use the following buildpack: https://github.com/sreid/heroku-buildpack-sourceversion. Support for this buildpack is built in.

Alternatively, you can set the COMMIT_HASH environment variable.