5.7.2 • Published 6 years ago

github-task-manager v5.7.2

Weekly downloads
4
License
MIT
Repository
github
Last release
6 years ago

github-task-manager

npm version Build Status Code Climate Test Coverage Greenkeeper badge Commitizen friendly semantic-release Docker Build Status

receive github hook, notify agent, receive task results, notify github (Unofficial)

Aims

Create an asynchronous CI agnostic mechanism for running custom test stage gates for github pull requests.

  • allow team leads to configure task sequences without leaving github
  • allow developers to see output from tasks without leaving github
  • trigger multiple jobs in parallel and indicate pending status on pr checks
  • then add results for each back to pull request check/comments as they complete
  • make extensible for other github event handling
  • stateless and deployable to Kubernetes at scale

Design

  • Deploy two functions to lambda via serverless framework ('gtmGithubHook', 'gtmGithubResults')
  • github PR open hook pointed at 'gtmGithubHook' lambda which adds event to a 'PendingQueue' SQS queue
  • agent(s) watch SQS for new test execution jobs
  • agent notifies 'ResultsQueue' SQS queue that a task has started (lambda updates github PR state)
  • agent triggers CI PR build, deploy and tests (plugin for each CI type)
  • agent formats and adds test results to 'ResultsQueue' SQS queue
  • agent notifies SNS topic which triggers 'gtmGithubResults' lambda
  • 'gtmGithubResults' lambda posts results to github pull request.

flow

Install

Environment variabledescription
GTM_AWS_KMS_KEY_IDaws kms key id
GTM_CRYPT_GITHUB_TOKENencrypted access token for accessing github
GTM_CRYPT_GITHUB_WEBHOOK_SECRETencrypted shared secret from github webook config
GTM_CRYPT_AWS_ACCESS_KEY_IDencrypted aws key id - for agent only
GTM_CRYPT_AWS_SECRET_ACCESS_KEYencrypted aws secret - for agent only
GTM_CRYPT_AGENT_AWS_SECRET_ACCESS_KEYsecret key for agent
GTM_CRYPT_AGENT_AWS_ACCESS_KEY_IDaccess key for agent
GTM_CRYPT_JENKINS_TOKENencrypted token
GTM_CRYPT_TEAMCITY_PASSCODEencrypted teamcity executor passcode
GTM_CRYPT_SONAR_LOGINencrypted sonar access token
GTM_CRYPT_SONAR_GITHUB_OAUTHencrypted github token for sonar to post comments and status
GTM_CRYPT_DOCKER_REG_PASSWORDencrypted docker private registry password
GTM_AWS_REGIONawsregion to create resources in
GTM_SQS_PENDING_QUEUEname of SQS queue for new event
GTM_SQS_RESULTS_QUEUEname of SQS queue for results
GTM_SNS_RESULTS_TOPICname of SNS topic for result ping
GTM_GITHUB_HOSTapi hostname can be updated for github enterprise
GTM_GITHUB_DEBUGdebug mode for api calls
GTM_GITHUB_TIMEOUTgithub api timeout
GTM_GITHUB_PATH_PREFIXpath prefix for github enterprise
GTM_GITHUB_PROXYgithub api client proxy
GTM_TASK_CONFIG_FILENAMEfilename in repo to look for for task config - default is .githubTaskManager
AWS_PROXYURL of proxy to use for network requests. Optional
GTM_AGENT_PORTdefaults to 9091
GTM_JENKINS_USERlogin for jenkins executor
GTM_JENKINS_URLurl executor uses to talk to jenkins
GTM_JENKINS_CSRFis csrf enabled? true or false
GTM_TEAMCITY_USERteamcity executor user
GTM_TEAMCITY_URLteamcity api url
GTM_DOCKER_IMAGE_WHITELISTcomma separated list of regex of allows docker images eg. alpine:*,bash:latest
GTM_DOCKER_IMAGE_WHITELIST_FILEuse an optional docker whitelist file .dockerImageWhitelistExample
GTM_DOCKER_COMMANDS_ALLOWEDdefault is false, set to true to enable docker executor
GTM_DOCKER_ALLOW_PULLallow agent to pull from registry
GTM_DOCKER_DEFAULT_WORKER_IMAGEfor running ci tasks, default is zotoio/gtm-worker:latest
GTM_DOCKER_REG_USERNAMEusername for docker private registry
GTM_DOCKER_REG_SERVERhostname for docker private registry
IAM_ENABLEDagent host uses IAM ?
LAUNCHDARKLY_API_TOKENtoken for launchdarkly sass executor
GTM_LOGSTASH_HOSToptional logstash host for elasticsearch analysis
GTM_LOGSTASH_PORToptional logstash port
GTM_SONAR_HOST_URLsonar host url to connect to
GTM_SONAR_PROJECTNAME_PREFIXprefix if reporting to sonarqube
GTM_SONAR_ANALYSIS_MODEmode for sonar runner, default preview for PRs
GTM_SONAR_SOURCESdefault source dir is src
GTM_SONAR_JAVA_BINARIESdefault is target
GTM_SONAR_MODULEScomma separated modules
GTM_SONAR_GITHUB_ENDPOINToptional enterprise github api url
GTM_TASK_CONFIG_DEFAULT_URLurl to default sample config used when repo is missing .githubTaskManager.json
GTM_TASK_CONFIG_DEFAULT_MESSAGE_PATHpath to markdown comment file added to PRs when repo is missing .githubTaskManager.json
GTM_DYNAMO_TABLE_EVENTSDynamoDB table to store event summaries
GTM_DYNAMO_TABLE_AGENTSDynamoDB table to store agent summaries
GTM_AWS_VPC_IDvpc id - only required for ddb endpoints
GTM_BASE_URLBase url used to render links to agent ui - eg elb cname
GTM_WELCOME_MESSAGE_ENABLEDIf not 'false', send a warning message on unconfigured repository pull requests
GTM_S3_DEPENDENCY_BUCKETaws s3 storage of build dependencies
GTM_AWS_S3_PROXYhttps_proxy for aws s3
GTM_REPO_BLACKLISTcomma separated list of regex to blackist repo names from triggering events
GTM_SLS_EXECUTOR_AWS_STAGEstage override from default calculation of dev/test
GTM_SLS_EXECUTOR_AWS_REGIONaws region for lambdas default ap-southeast-2
GTM_SLS_EXECUTOR_AWS_EXECUTION_ROLEdocker serverless lambda execution role
GTM_SLS_EXECUTOR_DEPLOY_MODEdeploy multiple lambdas 'parallel' (default) or 'sequential'
GTM_SLS_EXECUTOR_SNS_ERROR_TOPIC_ARNsns topic to notify on serverless error
GTM_SLS_EXECUTOR_APIGW_ENDPOINT_TYPEEDGE or REGIONAL or PRIVATE
GTM_SLS_EXECUTOR_APIGW_DOMAIN_SUFFIXreverse proxy domain name that will have apiId added as subdomain. eg. lambda.mysuffix.com will result in apiId.lambda.mysuffix.com
GTM_SLS_EXECUTOR_HTTP_PROXYproxy passed to serverless executor
GTM_SLS_EXECUTOR_NO_PROXYno_proxy passed to serverless executor
GTM_SLS_EXECUTOR_VPC_IDvpc id for private apigw endpoints
GTM_SLS_EXECUTOR_VPC_SECURITY_GROUP_IDvpc security group id
GTM_SLS_EXECUTOR_VPC_SUBNET_Avpc az subnet
GTM_SLS_EXECUTOR_VPC_SUBNET_Bvpc az subnet
GTM_SLS_EXECUTOR_VPC_SUBNET_Cvpc az subnet
GTM_SLS_EXECUTOR_AWS_KMS_KEY_IDkms key id for sls env var encryption
GTM_SLS_EXECUTOR_CONFIG_TYPE'ssm' (aws parameter store) or 'spring' or 'dotenv'
GTM_SLS_EXECUTOR_SPRING_CONFIG_ENDPOINTendpoint url of spring config server eg. http://spring:8888 if GTM_SLS_EXECUTOR_CONFIG_TYPE is 'spring'
GTM_WORKER_SCRIPTS_CLONEfor docker executors using https://github.com/zotoio/gtm-worker based image - url of git repo to overlay on workspace eg. https://github.com/zotoio/gtm-worker-scripts.git
GTM_WORKER_SCRIPTS_PATHdirectory within scripts clone repo to overlay

important: values of env vars prefixed with GTM_CRYPT_* must be created via npm run sls-encrypt [name] [value]

Configure and deploy

  • run: npm run sls-deploy - note that this will create aws re$ources..
  • capture the hook url output in console and add to github repo pull request conf
  • run: npm run sls-logs-hook or npm run sls-logs-results to tail the logs
  • create a .githubTaskManager.json in your repo per https://github.com/zotoio/github-task-manager/wiki/Creating-a-Task-Configuration
  • start an agent locally using npm run build && npm start agent (or use docker/k8s)
  • create a pull request and confirm the hook is being hit and agent processes event

Docker and Kubernetes agents

You can run the latest image from docker hub: https://hub.docker.com/r/zotoio/github-task-manager

npm run docker-hub-run

..or run using the local checkout and tail logs:

npm run docker-local-bounce

..or if you have a k8s cluster and kubectl configured:

npm run k8s-apply
npm run k8s-delete

note that these k8s npm script inject vars from .env into the manifest

k8s

a starting point k8s manifest is in ./k8s/k8s-gtm-agent.yml

Agent Configuration

Running the Agent

The agent uses environment variables to configure itself as well as any executors running within it. The below environment variables are required for basic operation of the GTM Agent.

TODO: Provide Variables

Agent Homepage

The GTM Agent provides an information page summarising the ongoing operation of the agent. The page is available on port 9091 by default.

Plugins

Task executors for Jenkins, Teamcity, Travis, Http, Docker are in progress. Custom task executors can be added by adding this project as a dependency, and registering new Executors and EventHandlers. Please see https://github.com/zotoio/gtm-agent for an example that you can fork and modify as required while still using this project as the core.

  • Executors contain the logic to run tasks against a given system type, and format the results.
  • EventHandlers are used to map Github events to specific functionality such as pull requests.

Contributing

Fork this repository and work on your enhancements, then send a pull request. If you build custom plugins that may be useful to others in your forked gtm-agent, please let us know and may be able to assist wwith backporting to this project.

Use commitizen for conventional commit messages via git cz instead of git commit.
To setup if not already installed:

npm install -g commitizen
npm install -g cz-conventional-changelog
echo '{ "path": "cz-conventional-changelog" }' > ~/.czrc

...or you can just use npm run commit which will use local commitizen install.

5.7.2

6 years ago

5.7.1

6 years ago

5.7.0

6 years ago

5.6.0

6 years ago

5.5.2

6 years ago

5.5.1

6 years ago

5.5.0

6 years ago

5.4.8

6 years ago

5.4.7

6 years ago

5.4.6

6 years ago

5.4.5

6 years ago

5.4.4

6 years ago

5.4.3

6 years ago

5.4.2

6 years ago

5.4.1

6 years ago

5.4.0

6 years ago

5.3.2

6 years ago

5.3.1

6 years ago

5.3.0

6 years ago

5.1.2

7 years ago

5.1.1

7 years ago

5.1.0

7 years ago

5.0.3

7 years ago

5.0.2

7 years ago

5.0.1

7 years ago

5.0.0

7 years ago

4.11.0

7 years ago

4.10.1

7 years ago

4.10.0

7 years ago

4.9.0

7 years ago

4.8.14

7 years ago

4.8.13

7 years ago

4.8.12

7 years ago

4.8.11

7 years ago

4.8.10

7 years ago

4.8.9

7 years ago

4.8.8

7 years ago

4.8.7

7 years ago

4.8.6

7 years ago

4.8.5

7 years ago

4.8.4

7 years ago

4.8.3

7 years ago

4.8.2

7 years ago

4.8.1

7 years ago

4.8.0

7 years ago

4.7.2

7 years ago

4.7.1

7 years ago

4.7.0

7 years ago

4.6.0

7 years ago

4.5.1

7 years ago

4.5.0

7 years ago

4.4.1

7 years ago

4.4.0

7 years ago

4.3.2

7 years ago

4.3.1

7 years ago

4.3.0

7 years ago

4.2.0

7 years ago

4.1.7

7 years ago

4.1.6

7 years ago

4.1.5

7 years ago

4.1.4

7 years ago

4.1.3

7 years ago

4.1.2

7 years ago

4.1.1

7 years ago

4.1.0

7 years ago

4.0.0

7 years ago

3.1.5

7 years ago

3.1.4

7 years ago

3.1.3

7 years ago

3.1.2

7 years ago

3.1.1

7 years ago

3.1.0

7 years ago

3.0.3

7 years ago

3.0.2

7 years ago

3.0.1

7 years ago

3.0.0

7 years ago

2.23.2

7 years ago

2.23.1

7 years ago

2.23.0

7 years ago

2.22.2

7 years ago

2.22.1

7 years ago

2.22.0

7 years ago

2.21.5

7 years ago

2.21.4

7 years ago

2.21.3

7 years ago

2.21.2

7 years ago

2.21.1

7 years ago

2.21.0

7 years ago

2.20.7

7 years ago

2.20.6

7 years ago

2.20.5

7 years ago

2.20.4

7 years ago

2.20.3

7 years ago

2.20.2

7 years ago

2.20.1

7 years ago

2.20.0

7 years ago

2.19.2

7 years ago

2.19.1

7 years ago

2.19.0

7 years ago

2.18.0

7 years ago

2.17.0

7 years ago

2.16.9

7 years ago

2.16.8

7 years ago

2.16.7

7 years ago

2.16.6

7 years ago

2.16.5

7 years ago

2.16.4

7 years ago

2.16.3

7 years ago

2.16.2

7 years ago

2.16.1

7 years ago

2.16.0

7 years ago

2.15.2

7 years ago

2.15.1

7 years ago

2.15.0

7 years ago

2.14.2

7 years ago

2.14.1

7 years ago

2.14.0

7 years ago

2.13.0

7 years ago

2.12.1

7 years ago

2.12.0

7 years ago

2.11.0

7 years ago

2.10.2

7 years ago

2.10.1

7 years ago

2.10.0

7 years ago

2.9.4

7 years ago

2.9.3

7 years ago

2.9.2

7 years ago

2.9.1

7 years ago

2.9.0

7 years ago

2.8.0

7 years ago

2.7.1

7 years ago

2.7.0

7 years ago

2.6.0

7 years ago

2.5.1

7 years ago

2.5.0

7 years ago

2.4.2

7 years ago

2.4.1

7 years ago

2.4.0

7 years ago

2.3.3

7 years ago

2.3.2

7 years ago

2.3.1

7 years ago

2.3.0

7 years ago

2.2.0

7 years ago

2.1.2

7 years ago

2.1.1

7 years ago

2.1.0

7 years ago

2.0.0

7 years ago

1.20.1

7 years ago

1.20.0

7 years ago

1.19.0

7 years ago

1.18.1

7 years ago

1.18.0

7 years ago

1.17.0

7 years ago

1.16.1

7 years ago

1.16.0

7 years ago

1.15.0

7 years ago

1.14.0

7 years ago

1.13.0

7 years ago

1.12.0

7 years ago

1.11.2

7 years ago

1.11.1

7 years ago

1.11.0

7 years ago

1.10.3

7 years ago

1.10.2

7 years ago

1.10.1

7 years ago

1.10.0

7 years ago

1.9.2

7 years ago

1.9.1

7 years ago

1.9.0

7 years ago

1.8.4

7 years ago

1.8.3

7 years ago

1.8.2

7 years ago

1.8.1

7 years ago

1.8.0

7 years ago

1.7.0

7 years ago

1.6.0

7 years ago

1.5.0

7 years ago

1.4.0

7 years ago

1.3.0

7 years ago

1.2.2

7 years ago

1.2.1

7 years ago

1.2.0

7 years ago

1.1.1

7 years ago

1.1.0

7 years ago

1.0.4

7 years ago

1.0.3

7 years ago

1.0.2

7 years ago

1.0.1

7 years ago

1.0.0

7 years ago