All Projects → kevinjqiu → Pat

kevinjqiu / Pat

Licence: apache-2.0
Prometheus Alert Testing utility

Programming Languages

go
31211 projects - #10 most used programming language

Projects that are alternatives of or similar to Pat

Example Prometheus Nodejs
Prometheus monitoring example with Node.js
Stars: ✭ 249 (+59.62%)
Mutual labels:  prometheus, alerting
Unsee
Alert dashboard for Prometheus Alertmanager
Stars: ✭ 700 (+348.72%)
Mutual labels:  prometheus, alerting
Netdata
Real-time performance monitoring, done right! https://www.netdata.cloud
Stars: ✭ 57,056 (+36474.36%)
Mutual labels:  prometheus, alerting
alertmanager-irc-relay
Send Prometheus Alerts to IRC using Webhooks
Stars: ✭ 44 (-71.79%)
Mutual labels:  alerting, prometheus
Prometheus
The Prometheus monitoring system and time series database.
Stars: ✭ 40,114 (+25614.1%)
Mutual labels:  prometheus, alerting
Awesome Prometheus Alerts
🚨 Collection of Prometheus alerting rules
Stars: ✭ 3,323 (+2030.13%)
Mutual labels:  prometheus, alerting
Hastic Server
Hastic data management server for analyzing patterns and anomalies from Grafana
Stars: ✭ 292 (+87.18%)
Mutual labels:  prometheus, alerting
Alertmanager2es
Receives HTTP webhook notifications from AlertManager and inserts them into an Elasticsearch index for searching and analysis
Stars: ✭ 173 (+10.9%)
Mutual labels:  prometheus, alerting
Ciao
HTTP checks & tests (private & public) monitoring - check the status of your URL
Stars: ✭ 1,322 (+747.44%)
Mutual labels:  prometheus, alerting
Ansible Alertmanager
Deploy Prometheus Alertmanager service
Stars: ✭ 93 (-40.38%)
Mutual labels:  prometheus, alerting
Karma
Alert dashboard for Prometheus Alertmanager
Stars: ✭ 1,007 (+545.51%)
Mutual labels:  prometheus, alerting
Prom2teams
prom2teams is an HTTP server built with Python that receives alert notifications from a previously configured Prometheus Alertmanager instance and forwards it to Microsoft Teams using defined connectors
Stars: ✭ 122 (-21.79%)
Mutual labels:  prometheus, alerting
Grafana
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
Stars: ✭ 45,930 (+29342.31%)
Mutual labels:  prometheus, alerting
Hollowtrees
A ruleset based watchguard to keep spot/preemptible instance based clusters safe, with plugins for VMs, Kubernetes, Prometheus and Pipeline
Stars: ✭ 141 (-9.62%)
Mutual labels:  prometheus, alerting
Php Fpm Exporter
Prometheus exporter for php-fpm status.
Stars: ✭ 146 (-6.41%)
Mutual labels:  prometheus
Android Client
An android client for the MifosX platform
Stars: ✭ 150 (-3.85%)
Mutual labels:  unit-testing
Elephant Shed
PostgreSQL Management Appliance
Stars: ✭ 146 (-6.41%)
Mutual labels:  prometheus
Promster
⏰A Prometheus exporter for Hapi, express and Marble.js servers to automatically measure request timings 📊
Stars: ✭ 146 (-6.41%)
Mutual labels:  prometheus
Jmx exporter
A process for exposing JMX Beans via HTTP for Prometheus consumption
Stars: ✭ 2,134 (+1267.95%)
Mutual labels:  prometheus
Runtimeunittesttoolkit
CLI/GUI Frontend of Unity Test Runner to test on any platform.
Stars: ✭ 150 (-3.85%)
Mutual labels:  unit-testing
 ____   _  _____ 
|  _ \ / \|_   _|
| |_) / _ \ | |  
|  __/ ___ \| |  
|_| /_/   \_\_|  
                 

Prometheus Alert Testing tool

CircleCI

You may also be interested in PromCLI

Build & Install

go get github.com/kevinjqiu/pat

You must have golang 1.9+ and dep installed.

Build from source

Check out this repo to $GOPATH/src/github.com/kevinjqiu/pat

and then:

cd $GOPATH/src/github.com/kevinjqiu/pat && make build

Usage

pat [options] <test_yaml_file_glob>

e.g.,

pat test/*.yaml

Test File Format

Test files are written in yaml format. For a complete schema definition (in jsonschema format), see here.

Top level attributes

  • name - The name of the test case
  • rules - The rule definitions that are under test
  • fixtures - The fixture setup for the tests
  • assertions - The test assertions

Rules

The rules section defines how the rules-under-test should be loaded. Currently, two rules loading strategies are supported:

  • fromFile - load the rules from a .rules yaml file. If the path specified is not an absolute path, the rule file path will be relative to the test file.
  • fromLiteral - embed the rules under test right inside the test file.

Example

rules:
  fromFile: http-rules.yaml

or

rules:
  fromLiteral: |-
    groups:
      - name: prometheus.rules
        rules:
          - alert: HTTPRequestRateLow
            expr: http_requests{group="canary", job="app-server"} < 100
            for: 1m
            labels:
              severity: critical

Fixtures

The fixtures section defines a list of metrics fixtures that the tests will be using. Each item in the list has the following attributes:

  • duration - How long these metrics will be set to the specified value. The duration must be acceptable by Golang's time.ParseDuration(), e.g., 5m (5 minutes), 1h (1 hour), etc.
  • metrics - The metrics and their values

Example

fixtures:
  5m:
    - http_requests{job="app-server", instance="0", group="blue"}	75
    - http_requests{job="app-server", instance="1", group="blue"}	120

This will create these two metrics, with the values last for 5 minutes.

You are also able to specify multiple metrics values:

  5m:
    - http_requests{job="app-server", instance="0", group="blue"}	75 100 200

In this case, the metric http_requests{job="app-server", instance="0", group="blue"} will be set to 75 for the first 5 minutes, 100 for the next 5 minutes and 200 for the next 5 minutes. You can use this form to easily setup long running time series.

Assertions

The assertions section contains a list of expectations when the alert rules are evaluated at certain time.

  • at - The instant when the rules are being evaluated
  • expected - The list of expected alert properties

Example

assertions:
  - at: 0m
    expected:
      - alertname: HTTPRequestRateLow
        alertstate: pending
        job: app-server
        severity: critical
  - at: 5m
    expected:
      - alertname: HTTPRequestRateLow
        alertstate: firing
        job: app-server
        severity: critical
  - at: 10m
    expected: []

In this example, we're asserting that when the alert rules are evaluated at 0m, with the given fixtures, we should get HTTPRequestRateLow alert in pending state, and when evaluated at 5m, the alert should be in firing state. When evaluated at 10m, we shouldn't get any alert.

A Complete Example

Suppose you have the following rule file that you want to be tested:

groups:
  - name: prometheus.rules
    rules:
      - alert: HTTPRequestRateLow
        expr: http_requests{group="canary", job="app-server"} < 100
        for: 1m
        labels:
          severity: critical

Write a yaml file with your test cases:

name: Test HTTP Requests too low alert
rules:
  fromFile: rules.yaml
fixtures:
  - duration: 5m
    metrics:
      - http_requests{job="app-server", instance="0", group="canary", severity="overwrite-me"}	75 85  95 105 105  95  85
      - http_requests{job="app-server", instance="1", group="canary", severity="overwrite-me"}	80 90 100 110 120 130 140
assertions:
  - at: 0m
    expected:
      - alertname: HTTPRequestRateLow
        alertstate: pending
        group: canary
        instance: "0"
        job: app-server
        severity: critical
      - alertname: HTTPRequestRateLow
        alertstate: pending
        group: canary
        instance: "1"
        job: app-server
        severity: critical
    comment: |-
      At 0m, the alerts met the threshold but has not met the duration requirement. Expect the alert to be pending
  - at: 5m
    expected:
      - alertname: HTTPRequestRateLow
        alertstate: firing
        group: canary
        instance: "0"
        job: app-server
        severity: critical
      - alertname: HTTPRequestRateLow
        alertstate: firing
        group: canary
        instance: "1"
        job: app-server
        severity: critical
    comment: |-
      At 5m, the alerts should be firing because the duration requirement is met.
  - at: 10m
    expected:
      - alertname: HTTPRequestRateLow
        alertstate: firing
        group: canary
        instance: "0"
        job: app-server
        severity: critical
    comment: |-
      At 10m, the alert should be firing only for instance 0 because instance 1 is >= 100.
  - at: 15m
    expected: []
    comment: |-
      At 15m, both instances are back to normal, therefore we expect no alert.

Run the test:

$ ./pat examples/test.yaml
=== RUN   Test_HTTP_Requests_too_low_alert_at_0m
--- PASS: Test_HTTP_Requests_too_low_alert_at_0m (0.00s)
=== RUN   Test_HTTP_Requests_too_low_alert_at_5m
--- PASS: Test_HTTP_Requests_too_low_alert_at_5m (0.00s)
PASS
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].