opinionated-ci-pipeline v2.0.1

Opinionated CDK CI Pipeline

CI/CD utilizing CDK Pipelines.

Features:

• pipeline deploying application from the default branch to multiple environments on multiple accounts,
• feature branch deployments to ephemeral environments,
• development environments deployments from the local CLI,
• build status notifications to repository commits,
• build failures notifications to SNS.

Currently supported source repositories are GitHub and Bitbucket.

Table of contents

• Table of contents
• Usage
  • 1. Install
  • 2. Set context parameters
  • 3. Create CDKApplication
  • 4. Create repository access token secret
    • GitHub
    • Bitbucket
  • 5. Bootstrap the CDK
  • 6. Deploy the CI Stack
  • 7. Setup source repository mirroring
    • GitHub
    • Bitbucket
  • Deploy development environment
• Parameters
• Notifications and alarms
• Library development

Usage

To set up, you need to complete the following steps:

1. Install the library in your project.
2. Specify target account and region for deployed environments.
3. Create CDKApplication with build process configuration.
4. Create repository access token for build status notifications.
5. Bootstrap the CDK on the AWS account(s).
6. Deploy the CI.
7. Setup source repository mirroring to CodeCommit.

At the end, you will have CI pipeline in place, and be able to deploy your own custom environment from the CLI as well.

1. Install

For Node.js:

npm install -D opinionated-ci-pipeline

For Python:

pip install opinionated-ci-pipeline

2. Set context parameters

Add project name and environments config in the cdk.json as context parameters. Each environment must have account and region provided.

{
  "app": "...",
  "context": {
    "projectName": "myproject",
    "environments": {
      "default": {
        "account": "111111111111",
        "region": "us-east-1"
      },
      "prod": {
        "account": "222222222222",
        "region": "us-east-1"
      }
    }
  }
}

The project name will be used as a prefix for the deployed CI Stack name.

Environment names should match environments provided later in the CDKApplication configuration.

The optional default environment configuration is used as a fallback.

The CI pipeline itself is deployed to the ci environment, with a fallback to the default environment as well.

3. Create CDKApplication

In the CDK entrypoint script referenced by the cdk.json app field, replace the content with an instance of CDKApplication:

#!/usr/bin/env node
import 'source-map-support/register';
import {ExampleStack} from '../lib/exampleStack';
import {CDKApplication} from 'opinionated-ci-pipeline';

new CDKApplication({
    stacks: {
        create: (scope, projectName, envName) => {
            new ExampleStack(scope, 'ExampleStack', {stackName: `${projectName}-${envName}-ExampleStack`});
        },
    },
    repository: {
        host: 'github',
        name: 'organization/repository',
    },
    packageManager: 'npm',
    pipeline: [
        {
            environment: 'test',
            post: [
                'echo "do integration tests here"',
            ],
        },
        {
            environment: 'prod',
        },
    ],
});

This configures the application with one Stack and a pipeline deploying to an environment test, running integration tests, and deploying to environment prod.

The test and prod environments will be deployed from the branch main (by default). All other branches will be deployed to separate environments. Those feature-branch environments will be destroyed after the branch is removed.

To allow deployment of multiple environments, the Stack(s) name must include the environment name.

4. Create repository access token secret

An access to the source repository is required to send build status notifications, visible in commit status and Pull Requests.

GitHub

Create a fine-grained personal access token with "commit statuses" read and write access.

Then create a secret in AWS Secrets Manager named /PROJECT_NAME/githubAuthorization with the generated token:

{
  "header": "Bearer xxx"
}

Bitbucket

Create an access token in Bitbucket repository settings with repository:write access.

Then create a secret in AWS Secrets Manager named /PROJECT_NAME/bitbucketAuthorization with the generated token:

{
  "header": "Bearer xxx"
}

5. Bootstrap the CDK

Bootstrap the CDK on the account holding the CI pipeline and all other accounts the pipeline will be deploying to.

When bootstrapping other accounts, add the --trust parameter with the account ID of the account holding the pipeline.

6. Deploy the CI Stack

Run:

cdk deploy -c ci=true

7. Setup source repository mirroring

Because of multiple drawbacks of "native" CodePipeline and CodeBuild integrations with GitHub and Bitbucket, the CI builds use CodeCommit as a source. You must configure GitHub / Bitbucket to mirror the repository to CodeCommit.

CI stack creates a CodeCommit repository (named the same as the projectName context value) and IAM user {PROJECT_NAME}-ci-repository-mirror-user with access to it.

GitHub

1. Create an SSH key pair with ssh-keygen -t rsa -b 4096 command.
2. Upload the public key as an SSH key for AWS CodeCommit in the created IAM user security credentials settings.
3. Create secrets in the GitHub repository settings under "Secrets and variables" -> "Actions":
   • CODECOMMIT_SSH_PRIVATE_KEY - the private key generated in step 1.
   • CODECOMMIT_SSH_PRIVATE_KEY_ID - the SSH key ID of uploaded key from IAM user.
4. Create .github/workflows/mirror-repository.yml file (update the REGION and PROJECT_NAME in target_repo_url):

name: Mirror to CodeCommit
on: [ push, delete ]
jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: pixta-dev/repository-mirroring-action@v1
        with:
          target_repo_url: ssh://git-codecommit.REGION.amazonaws.com/v1/repos/PROJECT_NAME
          ssh_private_key: ${{ secrets.CODECOMMIT_SSH_PRIVATE_KEY }}
          ssh_username: ${{ secrets.CODECOMMIT_SSH_PRIVATE_KEY_ID }}

Bitbucket

1. Create an SSH key pair in the Bitbucket repository Pipeline config.
2. Upload the public key as an SSH key for AWS CodeCommit in the created IAM user security credentials settings.
3. Create secrets in the Bitbucket repository Pipeline settings:
   • CODECOMMIT_SSH_PRIVATE_KEY - the private key generated in step 1.
   • CODECOMMIT_SSH_PRIVATE_KEY_ID - the SSH key ID of uploaded key from IAM user.
4. Put git-codecommit.REGION.amazonaws.com in Bitbucket repository SSH Keys Known hosts (update the REGION).
5. Create bitbucket-pipelines.yml file (update the REGION and PROJECT_NAME):

image: atlassian/default-image:4

clone:
  depth: full

pipelines:
  default:
    - step:
        name: Sync to CodeCommit
        script:
          - echo "User ${SSH_KEY_ID}" >> ~/.ssh/config
          # fetch all branches to mirror the whole repository
          - for remote in `git branch -r | grep -v -- '->' | grep -v -- "${BITBUCKET_BRANCH}"`; do git branch --track ${remote#origin/} $remote; done
          # mirror repository; it will remove deleted branches as well
          - git push --mirror ssh://git-codecommit.REGION.amazonaws.com/v1/repos/REPOSITORY-NAME

Deploy development environment

Run:

cdk deploy -c env=MYENV --all

to deploy arbitrary environments.

Parameters

• main pipeline:
  • preInstall
  • install
  • buildAndTest
  • synthPipeline
• feature branch environment deployment:
  • preInstall
  • install
  • buildAndTest
  • deployEnvironment
• feature branch environment destruction:
  • preInstall
  • install
  • destroyEnvironment

The location where CDK outputs synthetized files. Corresponds to the CDK Pipelines ShellStepProps#primaryOutputDirectory.

Notifications and alarms

Stack creates SNS Topics with notifications for main pipeline failures and feature branch build failures. Their ARNs are saved in SSM Parameters and outputed by the stack:

• main pipeline failures:
  • SSM: /{projectName}/ci/pipelineFailuresTopicArn
  • Stack exported output: {projectName}-ci-pipelineFailuresTopicArn
• feature branch build failures:
  • SSM: /{projectName}/ci/featureBranchBuildFailuresTopicArn
  • Stack exported output: {projectName}-ci-featureBranchBuildFailuresTopicArn

If you setup Slack notifications, you can configure those failure notifications to be sent to Slack.

Moreover, if you setup Slack notifications, an additional SNS Topic will be created to which you can send CloudWatch Alarms. It's ARN is provided:

• SSM: /{projectName}/ci/slackAlarmsTopicArn
• Stack exported output: {projectName}-ci-slackAlarmsTopicArn

Library development

Project uses jsii to generate packages for different languages.

Install dependencies:

npm install

Build:

npm run build

Install and deploy example application:

cd example
pnpm install
pnpm cdk deploy -c ci=true