@dcube-engineering/pipeline-specs-helper NPM

pipeline-specs-helper

Command Line Interface for easy generation of bamboo-specs yaml files from configuration json files.

Easily setup necessary files
Generate all the spec files for the project readily available to be used upon commit
Create specs files that tap into ready-made scripts available on pipeline-resources repository
Easily add extra custom scripts that are needed to suit specific project needs

More detailed documentation is available in the docs folder and listed here.

pipeline-specs-helper

Install

npm install --save-dev pipeline-specs-helper

To install this as a global command line tool, use

npm install -g pipeline-specs-helper

Usage

Initialise a repository to be used for bamboo-specs

npx pipeline init

Creates a base pipeline-specs-helper folder with a config file containing the bare minimum needed for a build plan. It also contains some basic pipeline scripts and also sample template files for stage customisation.

If there is already an existing folder and you would like to overwrite any conflicts, use:

npx pipeline init -f

To see in detail the files that were created, use:

npx pipeline init -v

Editing the config file

The full schema of the configuration file can be found here 1. Fill in project details (e.g. project name, plan key, plan name) 2. Add build agent requirements (docker build agent? host build agent? requirements?) 3. Add stages to the build taskSet (refer to list of stages and tasks for the full reference) 4. Reference the taskSet using the ref_taskSet property 5. Add variables to the plan that are needed by the stages (detailed in stages-and-tasks) 6. You have a finished config!

Adding Deployment or Scheduled Plans

Scheduled Plans are simply build plans with cron triggers. They can be used to define tasks that do not build artifacts for deployment but run parallel to the building process, like scanning environments etc.

They work similar to the build plan, just create a top level object called deployment or scheduled respectively
Add the stages you want to the deployment/scheduled taskSets
Create the deployment environment or the scheduled type objects under deployment.environments or scheduled.type
Repeat the same steps as you would for build plans

Generating the bamboo-spec yaml files

npx pipeline generate --type bamboo

This reads all the config files inside the top level of the pipeline-specs-helper folder and generates yaml spec files with filenames in the format '<plan-name>-<plan-type>-<any-extra-tags>-plan.yaml'. Bamboo spec files are written to the bamboo-specs directory automatically.

To generate only for specific config files, use

npx pipeline generate --type bamboo <config file 1> <config file 2> ...

The output name can be customised with the --outname flag. A common use would be to sort the output files into separate directories based on their project name, then plan name with a command like:

npx pipeline generate --type bamboo <name of config files> -o '%p/%n/%t-%g()%G-plan.yaml'

More information on the outname annotated string format can be found at the cli command page

Dealing with errors during generation

If errors occur during generation, an error will be output to the console. The errors are either errors during generation or errors during file writes.

Generation Errors

These errors normally occur if an error was encounted while parsing the config file to convert it to the yaml output. The error message will contain the config file in question, the plan that had the error, and the error that occurred. Examples of errors could be missing variables required by stages, or referencing a non-existent build agent, task set, or notification channel.

To fix the error, correct the mistake in the configuration file and run the generation command again.

WIP: Refer here for a list of error codes and the solutions.

Data for files parsed before the error was encounted will still be written to disk. Errors originating from one config file will not affect the generation on other configuration files. To see a list of all files that were successfully generated, use the -v flag.

Data write errors

These errors occur after a config file has been successfully converted to yaml output, but during file writing. Normally these would be due to an error in the file name. Ensure that the filename contained in the error message is a valid filename that can be written too. Change the outname parameter to a string that resolves to a valid filename. If the project name, plan name, or tags contains invalid characters, remove those.

Once the invalid characters have been removed, run the generation command again.

However, if the error message is unacceptable kind of an object to dump, there is likely an issue in the yaml output that has not been caught by the program. In such a case log a bug report.

If the error message is not helping in troubleshooting, try the troubleshooting self help guide.

Using custom templates

For customisation to more project specific usecases, custom templates can be written.

Templates need to accomplish two main tasks: 1. Export bamboo variables into specific environment variables. Using something like export ENV_VAR=${bamboo_VAR}; 2. Call pipeline-resources scripts, passing in an extra parameters to them that are not read from the environment. E.g. calling a script with ./path/to/script.sh ${bamboo_VAR1} -o ${bamboo_VAR2} -u ${bamboo_VAR3}

Details on writing templates can be found here

Templates inside the pipeline-specs-helper folder are automatically read and used in generation. Custom templates will take precedence over the default templates. If there are duplicates within the custom templates, the one specified first based on filename order will take precedence.

Templates inside stages/ are read as custom stage templates and likewise for jobs/ and deployment/

The deployment folder is for deployment tasks, used in deployment plans. More details are in the template tutorial.

WIP: using --template to specify custom templates
To use it for generating the yaml files, use any of the --template options. Annotated strings can be used to differentiate templates based on the project, plan, type and tags.
Example 1: build plans use the build-template.json, deployment plans use deployment-template.json etc.
npx pipeline generate --type bamboo --template '%t-template.json'
Example 2: build plans use custom.json, other plans use the default templates
npx pipeline generate --type bamboo --build-template 'custom.json'

Breaking Changes

When breaking changes are introduced to the tool, the major version will be bumped up. This signifies that old configuration files will be unlikely to work with the new version. A migration guide will be published, stating the migration steps in this format.

For moving of any properties (and all properties nested under them): (old key) -> (updated key)

For deprecating of old properties (old key) -> deleted, use (new key) instead

New stages that have been added (stage name): (variables and description)

Stages that have been changed (stage name): (changes)

Stages that are deprecated/removed (stage name): deprecated/removed; use (new stage name) instead