ci-cd_gitlab_aws v1.1.18

ci-cd_gitlab_aws

The ci-cd_gitlab_aws npm package is a small, opinionated continuous integration and deployment (ci-cd) library that is able to handle large complex projects and deploy development source, multiple test environments, and production-ready licensed artifacts.

The ci-cd_gitlab_aws npm deploys a set of scripts that can be invoked via gitlab-ci.yml, which begins a deployment on Gitlab, and in turn, deploys to AWS. The deployment to AWS includes a PKG Binary creation step, and depending on how the repo that uses this deployment is created, can be used to deploy either source of compiled binaries to AWS (and Docker). This project expects a child /bin submodule(s) where the latest binary is stored. This solution works very well with projects that contain nested submodules too Add an example project structure. As noted, only the latest PKG binary is ever at one time. The binary name is appended with the short commit number, which allows a number of projects to share a project as a submodule, and avoid unnecessary storage. The commit history for binary lists all available binaries, and they can be pulled by their short commit number for easy management.

This package does not contain any nodejs code, it is intended only as a platform for ci-cd deployment. It relies on a number of environment vars in the CI Section of Gitlab, AutoDevOps is enabled in Gitlab, a Personal Access Token for Gitlab, a few files present at the project level including modifications to package.json, and a VM(s) with base AMI configuration on AWS for deploy targets (see below). It will also create a binary package from the node.js code if a value is present for the BINARY_NAME environment variable.

The rest of this document provide detail on what is needed and provides conventions for way of working.

Check for updates to this NPM

You should do this regularly by running:

npm outdated | grep "ci-cd_gitlab_aws"

Updating to the newest deploy script

npm update ci-cd_gitlab_aws

Steps to add CI-CD_GITLAB_AWS to a project

• Make sure you have a git user specifically created for running scripts and automation. Do not use a personal account. This 'script' user will generate a Personal Access Token that will enable working securely with Gitlab.

• Enable AutoDevOps in your Gitlab project.

• Note your 'namespace'- are you an individual, an organisation, a group or a group with subgroups?

• Have a VM ready on AWS to which we'll deploy, and with a Elastic IP will help. Go to the bottom of this README to see the requirments for the "Base VM Image required".

• In Gitlab, go to “Settings -> CI/CD” and create these Environment Variables:

  DEPLOY_SERVERS_STAGING - comma separated ip values of all the servers to which you want to deploy this code

  DEPLOY_SERVERS_PROD - as above, for production

  PRIVATE_KEY - AWS SSH key. The key you downloaded when creating an EC2 instance

  TOKEN - A Personal Access Token created by a 'script' user; a user created primarily for the purpose of running scripts

  ENV_NAMESPACE - your user, groups or group/subgroup and repo name for Env vars. #eg: wyldtech%2Fenvfiles%2Fenv-files

  LINUX_USR - the user account required to SSH into

  AWS_DEFAULT_REGION Region to which you're deploying in AWS

  AWS_ACCESS_KEY_ID used by the AWS CLI to update the Security Group

  AWS_SECRET_ACCESS_KEY used by the AWS CLI to update the Security Group

  GROUP_NAME This is the Security Group Name that will be used to allow AWS SSH access for these scripts

  BINARY_NAME the name of the PKG binary to create. if empty, a binary will not be created (BIN_DIR and BIN_REPOPATH are irrelevant)

  BIN_DIR the name of the submodule directory

  BIN_REPOPATH the path and repo for the BIN_DIR submodule

• The deploy scripts also rely on a number of predefined env vars:

  USER=ubuntu - should be the same as LINUX_USR, but User isn't available in deploy.sh, only in update-start.sh
  HOME=/home/ubuntu - the users home directory on the deployment server
  CI_PROJECT_PATH=dlogic/npm-projects/testsci-cd_test - CI_PROJECT_NAMESPACE + CI_PROJECT_NAME
  CI_PROJECT_NAMESPACE=dlogic/npm-projects/tests - the full/group/subgroups path
  CI_PROJECT_NAME=ci-cd_test - the Repo name
  CI_ENVIRONMENT_NAME=test0 - this is used to find the ENV-files for each specific Environment
  CI_API_V4_URL=https://gitlab.com/api/v4 - the Gitlab version 4 API path used for cloning, etc.
  CI_COMMIT_SHORT_SHA=appended to the PKG binary filename

• This npm requires 3 config files (1 p) in the root of the implementing projects directory:

  • gitlab-ci.yml - this is what kicks off the build if AutoDev ops is enabled. A few generic ones meet many needs an illustrate what can be done with gitlab-ci.yml

  • ecosystem.config.js enables PM2 to run and manage the deployed applications. A generic version of this file should work in the majority of cases.

  • package.json EXISTING - this generally already exists. Below we show the modifications needed.

• update your package.json by adding this dependancy: `"ci-cd_gitlab_aws": "^1.0.0"

• update your package.json by adding this scripts: “deploy”: “cd ./node_modules/ci-cd_gitlab_aws/ && ./bin/deploy.sh”,

• In .gitlab-ci.yml, deploy.sh can be called with 2 switches, (-PROD or -0) without a switch argument.

  • -PROD is called for Production Deployment Servers, otherwise the Staging Deployment Servers will be used.

  • -0 is used to skip uploading environment files to the deployment VMs.

.gitlab-ci.yml

# Node docker image on which our code would run
image: node:10.0.0

#This command is run before all the jobs
before_script:
   #- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  - npm install

stages:
  - test
  - deploy

# lint and test are two different jobs in the same stage.
# This allows us to run these two in parallel and making build faster
# Job 2:
test:
  stage: test
  script:
    - npm run test

# Deploy
deploy_test0:
  only:
    - test
  stage: deploy
  script:
    - npm run deploy #can also add the -PROD switch- need to test
  environment:
    name: test0
    url: https://test0.XXXXXX.net

• Add the ecosystem.config.js required by the script to Start/Restart PM2, in the root of the project directory.

• Change the apps name in ecosystem.config.js: from ci-cd_test' to your repo name

.ecosystem.config.js

module.exports = {
  apps : [{
    name: 'ci-cd_test',
    script: 'app.js',

    // Options reference: https://pm2.io/doc/en/runtime/reference/ecosystem-file/
    args: 'one two',
    instances: 1,
    autorestart: true,
    watch: false,
    max_memory_restart: '1G',
    env: {
      NODE_ENV: 'development'
    },
    env_production: {
      NODE_ENV: 'production'
    }
  }],

  deploy : {
    production : {
      user : 'node',
      host : '212.83.163.1',
      ref  : 'origin/master',
      repo : 'git@github.com:repo.git',
      path : '/var/www/production',
      'post-deploy' : 'npm install && pm2 reload ecosystem.config.js --env production'
    }
  }
};

• The main entry point should be app.js in the root of the project directory. This is a convention; you can place it elsewhere if you tell PM2 where to start it, in ecosystem.config.js, or create a symlink to the different location.

• this NPM also requires a gitlab repo for pulling Env Files. It pulls from the master branch only, and expects subdirectories in the repo- named after each Environment defined. Capitalisation counts.

Base VM Image required

We have primarily tested using an Ubuntu instance, and create a base AMI on AWS with the following minimum spec:

• Ubuntu Server LTS 14.04 T2.micro

• NVM, Node, NPM, PM2 and Express pre-installed

• We're manually added 'PermitUserEnvironment yes' to etc/ssh/sshd_config and restarted the VM.