@eyeswild/cloudconfig NPM

CloudConfig

Opinionated configuration and deployment for node.js projects on Google Cloud App Engine. Inspired by @invisible/deploy. Supports both dotenv and node-config for loading configuration.

Development is currently paused whilst we're working on a project with serverless architecture.
Latest npm package

Master branch

This is an alpha version still under heavy development. Some parts do not yet work and there might be breaking changes each patch version until v1.0.0. Dotenv and node-config haven't yet been wrapped to support transparent loading of config files from storage buckets alongside deployed config files in a running Google App Engine service instance. Files may still be locally synced and deployed alongside services.

Installation and use

Install Google Cloud tools ensuring gcloud and gsutil are on your path Copy Scripts into your package.json Run npm i @eyeswild/cloudconfig Run npm run setup:install

A rough guide to setting up your project

Create the following projects in Google Cloud Console:
projectname, cloud storage for master config repository and localhost configuration buckets
projectname-<environment>, for app configuration and running app service on the app engine instance, e.g.
projectname-development
projectname-staging
projectname-productionetc.
Check that you have write permissions to the new projects - billing must be enabled to allow creating storage buckets.
Copy scripts from below into your package.json, changing start scripts as needed for your app
Run npm run setup:storage to create the necessary minimum configuration for cloudconfig in your project
Add your app's configuration to the relevant dotenv .env or node-config .json files. storage/cloudconfig shouldn't contain any sensitive configuration as it is required for localhost use.
Search & replace to change projectname to your own app project name across all files in storage, and in cloudconfig.env
Add the storage folder to your .gitignore
Run a build then commit cloudconfig.env, and config folders
Run npm run setup:cloud to create config storage buckets in the correct cloud projects
Check that per-user permissions are correctly set on the storage buckets for whatever access control you desire for e.g. restricting developer access to production credentials
To deploy to an environment, e.g. production:
Check that the App Engine configuration file for the environment is correct storage/cloudconfig/.env-<environment>.yaml
Run npm run cfg:push:production to sync config to the correct Google Cloud project and storage bucket for the environment
Run npm run deploy:production to create and deploy an instance of your app in that environment.

To support syncing a localhost configuration between developers: NOT YET FINISHED

Create a new service user for cloudconfig, e.g. cloudconfig@projectname.iam.gserviceaccount.com
Grant the service user Storage Legacy Bucket/Object Reader/Writer permissions as appropriate to enable pull/push of localhost configuration
Download or add service user credentials to storage/cloudconfig-service-account.json and commit
Other developers may update from repo and run npm run setup:developer

Push all configuration to repository

npm run cfg:push:master

Push relevant configuration to an environment

npm run cfg:push:<environment>, e.g: npm run cfg:push:development

Add config sync to your app code

import * as CloudConfig from '@eyeswild/cloudconfig'
import {createLogger} from 'winston'
import TODO from '@xdc/todo'

...

const log = createLogger({ ..... })    

...

// sync configuration overrides from appropriate google bucket 
// for project, environment, and service; for dotenv and 
// node-config configuration
CloudConfig.sync({
  logger: log,    // optional
  streamConfig: true
}).subscribe({
  next: (cfg: CloudConfig.Configuration) => {
    log.info('received configuration: ' + util.format(cfg))
  },
  complete: () => {
    logger.info('fetched app configuration, starting server...')
    runServer()
  },
  error: (err: any) => {
    logger.error(`whilst fetching app configuration - ${err}`)
    process.exit(1)
  },
})

function runServer() {
  TODO() // etc...
}

NPM 'cli' scripts for syncing configuration

A roll-your-own cli - just copy and paste these scripts into your package.json!

These scripts are cross-platform but note that npx and cross-env are required on the path

npm i npx cross-env -g

Other global dependencies are optional but recommended to speed up npx:

npm i copy env-cmd markcat trash-cli -g

"help": "npx markcat README-CLI.md",
"build": "tsc",
"clean": "npx trash compiled",
"empty-trash": "npx empty-trash",
"prestart": "npm run build",
"prebuild": "npm run clean",
"postbuild": "npm run cloudconfig:build",
"postclean": "npm run cloudconfig:clean",
"setup:storage": "npm i npx && npx copy \"./node_modules/@eyeswild/cloudconfig/storage/*\" ./storage && npx copy ./node_modules/@eyeswild/cloudconfig/cloudconfig.env . && mkdir env",
"setup:cloud": "npm run bucket:mk:root && npm run bucket:mk:all",
"setup:developer": "npm run setup:install && mkdir storage && mkdir env && npx env-cmd ./cloudconfig.env npm run cfg:pull:localhost",
"setup:install": "npm i -g npx cross-env env-cmd trash-cli empty-trash-cli copy && npm i && npm i -D",
"start": "npm run start:cloud",
"start:cloud": "node -r dotenv/config compiled/index.js dotenv_config_path=./dot.env",
"start:localhost": "npm run build && npx cross-env-shell NODE_ENV=localhost GOOGLE_APPLICATION_CREDENTIALS=storage/cloudconfig-service-account.json npm run start:cloud",
"app:deploy:production": "npx env-cmd ./storage/production/env/dot.env npm run util:app:deploy",
"cfg:push:all": "npm run cfg:push:development && npm run cfg:push:production && npm run cfg:push:master",
"cfg:push:localhost": "npx env-cmd ./storage/localhost/env/dot.env npm run util:cfg:push",
"cfg:push:development": "npx env-cmd ./storage/development/env/dot.env npm run util:cfg:push",
"cfg:push:production": "npx env-cmd ./storage/production/env/dot.env npm run util:cfg:push",
"cfg:push:master": "npx env-cmd ./cloudconfig.env npm run sync:push:root",
"cfg:pull": "npx cross-env-shell \"npm run sync:pull:services && npm run sync:pull:dotenv\"",
"cfg:pull:localhost": "npx env-cmd ./cloudconfig.env npm run util:pull:localhost",
"cfg:pull:master": "npx env-cmd ./cloudconfig.env npm run rsync:pull:root",
"cloudconfig:build": "npx copy \"./storage/cloudconfig/env-*\" . && npx copy \"./storage/cloudconfig/config/*\" ./config && npx copy \"./storage/localhost/env/*\" .",
"cloudconfig:clean": "npx trash env-*.yaml && npx trash config && npx trash dot.env",
"bucket:mk:all": "npm run bucket:mk:localhost && npm run bucket:mk:development && npm run bucket:mk:production",
"bucket:mk:localhost": "npx env-cmd ./storage/localhost/env/dot.env npm run util:bucket:create",
"bucket:mk:development": "npx env-cmd ./stoyrage/development/env/dot.env npm run util:bucket:create",
"bucket:mk:production": "npx env-cmd ./storage/production/env/dot.env npm run util:bucket:create",
"bucket:mk:root": "npx env-cmd ./cloudconfig.env npm run util:bucket:create",
"bucket:rm:all": "npm run bucket:rm:localhost && npm run bucket:rm:development && npm run bucket:rm:production",
"bucket:rm:localhost": "npx env-cmd ./storage/localhost/env/dot.env npm run util:bucket:remove",
"bucket:rm:development": "npx env-cmd ./storage/development/env/dot.env npm run util:bucket:remove",
"bucket:rm:production": "npx env-cmd ./storage/production/env/dot.env npm run util:bucket:remove",
"sync:push:services": "npx cross-env gsutil -m rsync -d -r ./storage/$NODE_ENV/default $CONFIG_BUCKET",
"sync:push:dotenv": "npx cross-env gsutil -m cp -r ./storage/$NODE_ENV/env/dot.env $CONFIG_BUCKET/env/dot.env",
"sync:push:root": "npx cross-env gsutil -m rsync -r ./storage $CONFIG_BUCKET",
"sync:pull:services": "npx cross-env gsutil -m rsync -d -r $CONFIG_BUCKET/config ./storage/$NODE_ENV/default/config",
"sync:pull:dotenv": "npx cross-env gsutil -m cp -r $CONFIG_BUCKET/env/dot.env ./storage/$NODE_ENV/env/dot.env",
"sync:pull:root": "npx cross-env gsutil -m rsync -r $CONFIG_BUCKET ./storage",
"util:pull:localhost": "npx cross-env-shell NODE_ENV=localhost CONFIG_BUCKET=$CONFIG_BUCKET-localhost npm run cfg:pull",
"util:bucket:create": "npx cross-env gsutil mb -p $GCLOUD_PROJECT $CONFIG_BUCKET",
"util:bucket:remove": "npx cross-env gsutil rb $CONFIG_BUCKET",
"util:app:deploy": "npx cross-env-shell \"npm run build && npm run util:deploy:gcloud\"",
"util:deploy:gcloud": "npx cross-env gcloud app deploy ./env-$NODE_ENV.yaml --project=$GCLOUD_PROJECT",
"util:cfg:push": "npm run sync:push:services && npm run sync:push:dotenv"

Documentation still "TODO":

use with Compute Engine / non Google Cloud servers

Features still "TODO":

stream configuration into wrapped node-config objects to support read only AppEngine file system
move npm scripts into ccc.js cli
support for creating multiple localhost hostname configurations for dev teams
support service config versioning
support splitting environments using services rather than projects
...
cli tool to assist with manipulating and versioning configuration