dxp-console-iac v1.0.0

DXP Console IaC

Overview

The repository contains DXP Console backend API built on top of AWS services as well as CLI tool to work with AWS resources.

Structure

*
|_ auth0          : Auth0 related files
|_ cloudformation : Cloudformation files
|_ config         : Initial import data and other configs
|_ lambdas        : AWS Lambda functions
|_ dynamodb       : DynamoDB related files
|_ scripts        : CLI scripts to work with AWS resources and deployment
|_ test           : Test files

Requirements

• AWS CLI tool and credential to work with
• NodeJS / NPM

Installation

1. NPM installs required packages for deployment and testing

   $ npm install

2. Create .env file in the root folder

• Check the details for .env file here
• Pick a name for cloudformation stack to be used like skim
• Replace [NAME] varaible in the below doc with the name

3. Update the name of stage in the copied file with the name selected from step 2

4. Run npm link command on dxp-console lambda package

   $ cd lambdas/dxp-console;
   $ sudo npm link;

5. Create a new environment

   $ ./scripts/publish [NAME] --create

6. Publish the new environment

   $ ./scripts/publish [NAME] -f

7. Import the initial data

   $ ./scripts/publish [NAME] --import=rbac,organisations,tenants,users

Updating an existing environment

The existing cloudformation stack update can be done with one command

# Update CF
$ ./scripts/publish ENVIRONMENT_NAME

# Update CF as well as all lambda functions
$ ./scripts/publish ENVIRONMENT_NAME --function

# Update CF with selected lambda functions
$ ./scripts/publish ENVIRONMENT_NAME --function=InstanceCreate,JobSubmit

Local development

The NodeJS Express server simulates API Gateway and Lambda locally with DynamoDB running in container.

For the first time running:

$ npm run dynamodb:install
$ npm run dev:import

The command will reset DynamoDB container and import the initial data. After that you can start dev server like:

$ npm run dev:start

How it works

The publish script has similar features as Serverless framework. It helps to maintain cloudformation stack and update it with lambda functions.

Script Library

The script uses aws-sdk for NodeJS to interact with AWS. Currently, it uses S3 and Cloudformation libraries to upload lambdas and create/update stacks.

S3 storage for stage isolation

Each stage has its own S3 bucket to start. Each bucket has the following structure:

[STAGE]-dxp-console
|_ cloudformation
|  |_ 2019_12
|     |_ cf_[ISO_STRING_DATETIME].yml
|     |_ ...
|_ lambdas
   |_ 2019_[MONTH_IN_NUM]
     |_ [ISO_STRING_DATETIME]
         |_ instancecreate.zip
         |_ instanceget.zip
         |_ ...

On script execution, one ISO datetime string is created and used for CF Yml file and lambda function folder name. Note that even if only lambda functions got changed without CF change, the new CF YML file is uploaded to S3.

Cloudformation Parameters

When the script programmatically updates CF stack, it passes parameters JSON object to customise stack for each stage as well as set lambda function package file. The used parameters JSON file is also saved under cloudformation folder.

Lambda Functions

All lambda functions used in DXP Console is written in NodeJS. Each function exists as an independent node package under lambdas folder.

In order to support code sharing between different lambdas, a reserved dxp-console package exits and it is linked to all other lambda node packages via npm link command, which means that ff dxp-console code changes, all lambdas are advised to be re-published together.

Raw CLI Commands

These are aws CLI commands to work with AWS directly without using publish script.

# Upload parameters and CF Yml to S3
$ aws s3 cp ./cloudformation/dxp-console-cf.yml s3://dxp-console/cloudformation/prod-dxp-console-cf.yml
$ aws s3 cp ./cloudformation/prod-dxp-console-params.json s3://dxp-console/cloudformation/prod-dxp-console-params.json

# Cf update command
aws cloudformation update-stack --stack-name prod-dxp-console \
  --template-url https://prod-dxp-console.s3-ap-southeast-2.amazonaws.com/cloudformation/prod-dxp-console-cf.yml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameters file://cloudformation/prod-dxp-console-params.json

Old command to update API Doc to GCloud storage

$ redoc-cli bundle specs/dxp_api.yml --output specs/dxp_api.doc.html
$ gsutil cp ./specs/dxp_api.doc.html gs://demo.dxp.squiz.cloud/dev/dxp_api.doc.html

Local DynamoDB Command

$ docker run --rm -p 8000:8000 amazon/dynamodb-local