madflojo / Automatron

Automatron is a framework for creating self-healing infrastructure. Simply put, it detects system events & takes action to correct them.

The goal of Automatron is to allow users to automate the execution of common tasks performed during system events. These tasks can be as simple as sending an email to as complicated as restarting services across multiple hosts.

Features

• Automatically detect and add new systems to monitor
• Monitoring is executed over SSH and completely agent-less
• Policy based Runbooks allow for monitoring policies rather than server specific configurations
• Supports Nagios compliant health check scripts
• Allows dead simple arbitrary shell commands for both checks and actions
• Runbook flexibility with Jinja2 templating support
• Pluggable Architecture that simplifies customization

Runbooks

The core of Automatron is based around Runbooks. Runbooks are policies that define health checks and actions. You can think of them in the same way you would think of a printed runbook. Except with Automatron, the actions are automated.

A simple Runbook example

The below runbook is a very basic example, it will check if NGINX is running (every 2 minutes) and restart it after 2 unsuccessful checks.

name: Check NGINX
schedule: "*/2 * * * *"
checks:
  nginx_is_running:
    execute_from: target
    type: cmd
    cmd: service nginx status
actions:
  restart_nginx:
    execute_from: target
    trigger: 2
    frequency: 300
    call_on:
      - WARNING
      - CRITICAL
      - UNKNOWN
    type: cmd
    cmd: service nginx restart

The above actions will be performed every 300 seconds (5 minutes) until the health check returns an OK status. This delay allows time for NGINX to restart after each execution.

A complex Runbook with Jinja2

This next runbook example is a more complex version of the above. In this example we will use Jinja2 and Automatron's Facts to enhance our runbook further.

name: Check NGINX
{% if "prod" in facts['hostname'] %}
schedule:
  second: */20
{% else %}
schedule: "*/2 * * * *"
{% endif %}
checks:
  nginx_is_running:
    execute_from: target
    type: cmd
    cmd: service nginx status
actions:
  restart_nginx:
    execute_from: target
    trigger: 2
    frequency: 300
    call_on:
      - WARNING
      - CRITICAL
      - UNKNOWN
    type: cmd
    cmd: service nginx restart
  remove_from_dns:
    execute_from: remote
    trigger: 0
    frequency: 0
    call_on:
      - WARNING
      - CRITICAL
      - UNKNOWN
    type: plugin
    plugin: cloudflare/dns.py
    args: remove [email protected] apikey123 example.com --content {{ facts['network']['eth0']['v4'][0] }}

The above example uses Jinja2 and Facts to create a conditional schedule. If our target server has a hostname that contains the word "prod" within it. The schedule for the health check will be every 20 seconds. If not, it will be every 2 minutes.

Another addition is the remove_from_dns action, which will remove the target server's DNS entry using the CloudFlare DNS plugin.

By using Facts and Jinja2 together you can customize a single runbook to cover unique actions for multiple hosts and environments.

Stay in the loop

License

 Copyright 2016 Benjamin Cane

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.