GitHub IRC Notification Service

Ghi (pronounced 'ghee') is a relay between GitHub and IRC and/or Mastodon. It was created to take the place of the now depreciated GitHub IRC Service. Ghi receives events from GitHub for a specified repository via a webhook. Then it parses the event and sends the relevant information to your configured IRC channels and/or Mastodon timeline and/or Matrix rooms. Ghi was written to be very configuration driven. Therefore, Ghi is set up with a .ghi.yml file and can listen for multiple repositories and send to multiple IRC channels. Most of the features in the original GitHub Service are supported in Ghi as well.

Getting Started

Ghi was designed and written to be ran in AWS Lambda with API Gateway. However, I've also created a very simple HTTP server implementation so Ghi can be ran on any server if desired. Ghi is configured entirely with the .ghi.yml file. In this file you will set all necessary information including repositories, IRC nick, IRC host, channels, Mastodon instance, Mastodon user, Matrix homeserver, Matrix rooms, etc.

Deployment

AWS Lambda

Ghi was written to be ran in AWS Lambda and is the recommended deployment type. There are many ways to deploy Ghi to Lambda, I've found the simplest solution to be SAM. I've included several example SAM template in the examples/SAM.md file. These can get you started running Ghi in Lambda quickly, but your own deployment method will work as well. If using your own process, be sure to create an API Gateway as well. During the build process be sure install the dependencies in the requirements-aws.txt file and add your .ghi.yml file. Below is the necessary information for running in Lambda:

• runtime - python3.6
• handler - index.handler
• timeout - 75 seconds
• memory - 128 MB

Server

Ghi comes with a minimal HTTP server that can be used to run Ghi on a server if desired. If you need more advanced HTTP functionality like SSL, Error handling, load balancing, etc, you should use something like Nginx as a reverse proxy. To deploy, you can simply clone this repository or add Ghi to your existing deployment workflow. Ensure that during deployment you also bundle/install the dependencies in the requirements-server.txt file and add your .ghi.yml file.

$ git clone https://github.com/gkrizek/ghi.git
$ cd ghi/
$ pip3 install -r requirements-server.txt

Mastodon

Ghi supports pushing messages to Mastodon. Since Ghi, as the name implies, is mainly focused on IRC the module requirement for Mastodon is optional. If you want to use Mastodon as one of the outlets the Mastodon.py module is required:

$ pip3 install mastodon-py

Matrix

Ghi also supports pushing messages to Matrix. Again, since Ghi, as the name implies, is mainly focused on IRC the module requirement for Mastodon is optional. If you want to use Matrix as one of the outlets the matrix-nio module is required:

$ pip3 install matrix-nio

Setting Configuration

.ghi.yml

Location

The .ghi.yml file is what drives all the functionality of Ghi. When a request comes in, Ghi attempts to automatically find the .ghi.yml file from a list of common places in the filesystem. The directories it searches are (in order):

• Current Directory (./)
• Home Directory (~/)
• Temp Directory (/tmp)

You can also manually specify where the Ghi file is located with the GH_CONFIG_PATH environment variable. This will attempt to use the .ghi.yml file at the specified location regardless of any of the other file locations.

To explain, if I have a ~/.ghi.yml file and a ./.ghi.yml file in my current directory, Ghi will use the one in my current directory and ignore the one in my home directory. If I have GH_CONFIG_PATH="~/configs/.ghi.yml", then it will try to use that file only and ignore any others.

Contents

The Ghi file is where you specify things like repositories, branches, channels, and IRC/Mastodon/Matrix details. Ghi uses something called a "Pool" to determine which events do what. A Pool can have 1 or more repositories and 1 or more channels. You can also list multiple pools in a single Ghi instance. So you could have both gkrizek/repo1 and gkrizek/repo2 sending messages to #my-cool-channel while also having gkrizek/repo3 sending messages to #other-cool-channel and #last-cool-channel (and of course many more variations of that).

The top two required parameters of the Ghi file are version and pools. Currently there is only a version 1 of the Ghi file, but pools will be a list of Pool configurations. Each Pool is required to define some GitHub information like repository names and validation secrets. They will also need to specify IRC data like nick, host, password, and channels.

There are a lot more options that you can set to further configure Ghi. See the Configuration section.

Basic .ghi.yml Example:

version: 1
pools:
  - name: "my-pool"
    github:
      repos:
        - name: "gkrizek/repo1"
          secret: "3ccb8d36bd4c67dd1dffcff9ca2c40"
    irc:
      host: "chat.freenode.net"
      nick: "my-irc-bot"
      channels:
        - "my-cool-channel"
    mastodon:
      instance: "https://mstdn.social"
      user: "[email protected]"
      password: "myBotPassword123!"
      secretspath: "/home/thatsme/my/mastodonsecrets/"
      appname: "my-mastodon-bot"
    matrix:
      homeserver: "https://a.matrix.srv"
      user: "@ghibot:matrix.srv"
      password: "anotherGreatPassword456!"
      secretspath: "/home/thatsme/my/matrixsecrets/"
      device_id: "Ghi-Matrix-Bot"
      rooms:
        - "#room:matrix.srv"
    outlets:
      - "irc"
      - "mastodon"
      - "matrix"

More Ghi file examples in examples/.ghi.yml.md.

Important Note:

If you are running Ghi in AWS your IRC nick must be registered with an email and password. This is because Freenode requires all nicks connecting from AWS to be registered first to cut down on spam.

Environment Variable

Ghi allows for some configuration values to be set using environment variables. Primarily the values that are sensitive, such as the GitHub secret and IRC password.

To set the GitHub Secret with an environment variable rather than putting it in the .ghi.yml file, use the variable GHI_GITHUB_SECRET_<OWNER>_<REPO>. The <OWNER> and <REPO> are the owner and name of the repository with any special characters removed and all uppercase. So if you are trying to set a secret for repository gkrizek/ci-pipeline_1 you would use GHI_GITHUB_SECRET_GKRIZEK_CIPIPELINE1.

To set the password for your IRC bot with an environment variable, use GHI_IRC_PASSWORD_<NAME>. The <NAME> is the name you gave your pool in the .ghi.yml file with any special characters removed and all uppercase. So if I gave my pool a name of my-pool, I would set the IRC password with GHI_IRC_PASSWORD_MYPOOL.

GitHub

To configure GitHub to send events to your Ghi instance, you create a webhook in your repository. Here is the GitHub documentation on how to do that. The Payload URL is whatever URL you have given your Ghi instance. The path does not matter to Ghi. Ghi does not check the path of the request and will respond to any path. The Content Type should be application/json. It is highly recommended to use a Secret for the webhook, but not required. You can then select whichever Event Types you want Ghi to act on, however check that Ghi supports the event first.

Running

AWS Lambda

After you have deployed your function to Lambda, there isn't much else needed for running Ghi. Simply make requests to the provisioned endpoint and API Gateway/Lambda will handle the rest. When running in Lambda, the default behavior is to respond to the request instantly, then process the event. This is because the GitHub webhook expects a response in less than 10 seconds to be considered healthy. To accomplish this, we re-invoke the same lambda function again to process the event, then respond to the intial request. If you would like Ghi to only respond when the request is complete, you can set the environment variable GHI_LONG_RESPONSE="true". This will make Ghi appear as failing in GitHub, but should still be working properly. This could also be helpful for debugging.

Important Note:

If you are running Ghi in AWS your IRC nick must be registered with an email and password. This is because Freenode requires all nicks connecting from AWS to be registered first to cut down on spam.

Server

Once you have cloned or deployed Ghi to a server, you can run it by executing the ghi/server.py file. (Python 3 is required). This will start a web server listening on port 7890. Ghi sends all logs to stdout/stderr so redirecting (&>>) to a log file is recommended. You can change the listener port with either the --port flag or the GHI_PORT environment variable. Because GitHub required webhooks to respond within 10 seconds, the Ghi web server executes the function then responds right away.

Example:

$ python3 ghi/server.py --port 8080

Make sure your PYTHONPATH is set to your Ghi project directory.

I have provided an example service file to use with systemd to make starting/stopping Ghi easier. To use systemd's journal logging you can add the --systemd argument in the command line.

Configuration

.ghi.yml

Ghi is configurable and supports lots of combinations of repositories, channels, and pools. These are all possible configuration options and an example configuration with every value.

Top Level

Global Configuration Object

Pool Configuration Object

~ For all irc, mastodon, and matrix settings: if they're one of the configured outlets.

Repository Configuration Object

Example Configuration File

version: 1 # required
debug: true # optional, default False
global: # optional
    github: # optional
      shorten_url: true # optional, defaults to false
      verify: true # optional, defaults to true
    irc: # optional
      host: "chat.freenode.net" # optional, but must be set in pool if not here
      port: 6697 # optional, default is 6697 for ssl and 6667 for non-ssl
      ssl: true # optional, default is true
      nick: "my-irc-bot" # optional, but must be set in pool if not here
      password: "abc123" # optional, but must be set in pool if not here and if needed for the nick that's used
    mastodon: # optional
      instance: "https://mstdn.social" # optional, but must be set in pool if not here
      user: "[email protected]" # optional, but must be set in pool if not here
      password: "myBotPassword123!" # optional, but must be set in pool if not here
      secretspath: "/home/thatsme/my/mastodonsecrets/" # optional, but must be set in pool if not here
      appname: "my-mastodon-bot" # optional, but must be set in pool if not here
      merges_only: true # optional, default is true
    matrix: # optional
      homeserver: "https://a.matrix.srv" # optional, but must be set in pool if not here
      user: "@ghibot:matrix.srv" # optional, but must be set in pool if not here
      password: "anotherGreatPassword456!" # optional, but must be set in pool if not here
      secretspath: "/home/thatsme/my/matrixsecrets/" # optional, but must be set in pool if not here
      device_id: "Ghi-Matrix-Bot" # optional, default is "Ghi-Matrix-Bot"
    outlets: # optional, default is irc
      - "irc"
      - "mastodon"
      - "matrix"

pools: # required
  - name: "my-pool" # required
    github: # required
      repos: # required
        - name: "gkrizek/repo1" # at least 1 repo is required
          secret: "3ccb8d36bd4c67dd1dffcff9ca2c40" # optional, but if it's needed it must be set here or with environment variable
          branches: # optional, default is 'all'
            - "master"
            - "staging"
          verify: true # optional, default is true
      shorten_url: true # optional, defaults to false
    irc: # required if IRC is one of the outlets
      host: "chat.freenode.net" # required
      port: 6697 # optional, default is 6697 for ssl and 6667 for non-ssl
      ssl: true # optional, default is true
      nick: "my-irc-bot" # required
      password: "abc123" # optional, but if it's required by the nick it must be set here or with environment variable
      channels: # required
        - "my-cool-channel" # at least 1 channel is required
    mastodon: # required if Mastodon is one of the outlets
      instance: https://mstdn.social # required
      user: [email protected] # required
      password: myBotPassword123! # required
      secretspath: "/home/thatsme/my/mastodonsecrets/" # required
      appname: "my-mastodon-bot" # required
      merges_only: true # optional, default is true
    matrix: # required if Matrix is one of the outlets
      homeserver: "https://a.matrix.srv" # required
      user: "@ghibot:matrix.srv" # required
      password: "anotherGreatPassword456!" # required
      secretspath: "/home/thatsme/my/matrixsecrets/" # required
      device_id: "Ghi-Matrix-Bot" # required, default is "Ghi-Matrix-Bot"
      rooms: # required
        - "#room:matrix.srv" # at least 1 room is required
    outlets: # optional, default is irc
      - "irc"
      - "mastodon"
      - "matrix"

If you define a parameter in the Global section and in your pool, the value in the pool will be used.

See the examplesdirectory for .ghi.yml examples, as well as others.

Important Note:

If you are running Ghi in AWS your IRC nick must be registered with an email and password. This is because Freenode requires all nicks connecting from AWS to be registered first to cut down on spam.

Environment Variable

Some parameters are able to be exported as environment variables instead of being set in the Ghi file. These are all possible environment variables you can use.

Supported Event Types

• push
• pull_request