github-task-manager

receive github hook, notify agent, receive task results, notify github (Unofficial)

Aims

Create an asynchronous CI agnostic mechanism for running custom test stage gates for github pull requests.

• allow team leads to configure task sequences without leaving github
• allow developers to see output from tasks without leaving github
• trigger multiple jobs in parallel and indicate pending status on pr checks
• then add results for each back to pull request check/comments as they complete
• make extensible for other github event handling
• stateless and deployable to Kubernetes at scale

Design

• Deploy two functions to lambda via serverless framework ('gtmGithubHook', 'gtmGithubResults')
• github PR open hook pointed at 'gtmGithubHook' lambda which adds event to a 'PendingQueue' SQS queue
• agent(s) watch SQS for new test execution jobs
• agent notifies 'ResultsQueue' SQS queue that a task has started (lambda updates github PR state)
• agent triggers CI PR build, deploy and tests (plugin for each CI type)
• agent formats and adds test results to 'ResultsQueue' SQS queue
• agent notifies SNS topic which triggers 'gtmGithubResults' lambda
• 'gtmGithubResults' lambda posts results to github pull request.

Install

• clone this repo or npm install --save github-task-manager
• npm install
• setup serverless aws creds per https://github.com/serverless/serverless/blob/master/docs/providers/aws/guide/credentials.md
• setup a .env file in the repo root (copy from .envExample and modify)
• create and AWS KMS key, and capture the id for var GTM_AWS_KMS_KEY_ID

important: values of env vars prefixed with GTM_CRYPT_* must be created via npm run sls-encrypt [name] [value]

Configure and deploy

• run: npm run sls-deploy - note that this will create aws re$ources..
• capture the hook url output in console and add to github repo pull request conf
• run: npm run sls-logs-hook or npm run sls-logs-results to tail the logs
• create a .githubTaskManager.json in your repo per https://github.com/zotoio/github-task-manager/wiki/Creating-a-Task-Configuration
• start an agent locally using npm run build && npm start agent (or use docker/k8s)
• create a pull request and confirm the hook is being hit and agent processes event

Docker and Kubernetes agents

You can run the latest image from docker hub: https://hub.docker.com/r/zotoio/github-task-manager

npm run docker-hub-run

..or run using the local checkout and tail logs:

npm run docker-local-bounce

..or if you have a k8s cluster and kubectl configured:

npm run k8s-apply
npm run k8s-delete

note that these k8s npm script inject vars from .env into the manifest

a starting point k8s manifest is in ./k8s/k8s-gtm-agent.yml

Agent Configuration

Running the Agent

The agent uses environment variables to configure itself as well as any executors running within it. The below environment variables are required for basic operation of the GTM Agent.

TODO: Provide Variables

Agent Homepage

The GTM Agent provides an information page summarising the ongoing operation of the agent. The page is available on port 9091 by default.

Plugins

Task executors for Jenkins, Teamcity, Travis, Http, Docker are in progress. Custom task executors can be added by adding this project as a dependency, and registering new Executors and EventHandlers. Please see https://github.com/zotoio/gtm-agent for an example that you can fork and modify as required while still using this project as the core.

• Executors contain the logic to run tasks against a given system type, and format the results.
• EventHandlers are used to map Github events to specific functionality such as pull requests.

Contributing

Fork this repository and work on your enhancements, then send a pull request. If you build custom plugins that may be useful to others in your forked gtm-agent, please let us know and may be able to assist wwith backporting to this project.

Use commitizen for conventional commit messages via git cz instead of git commit.
To setup if not already installed:

npm install -g commitizen
npm install -g cz-conventional-changelog
echo '{ "path": "cz-conventional-changelog" }' > ~/.czrc

...or you can just use npm run commit which will use local commitizen install.