All Projects → ignatev → burrow-kafka-dashboard

ignatev / burrow-kafka-dashboard

Licence: GPL-3.0 license
Kubernetes Kafka Overview, Burrow consumer lag stats, Kafka disk usage

Projects that are alternatives of or similar to burrow-kafka-dashboard

Grafana Sync
Keep your Grafana dashboards in sync
Stars: ✭ 88 (+137.84%)
Mutual labels:  grafana, grafana-dashboard
idrac snmp-grafana
SNMP Based Dashboard to Monitor Dell Hosts via iDRAC
Stars: ✭ 88 (+137.84%)
Mutual labels:  grafana, grafana-dashboard
Pi Hole Influx
A python daemon to send Pi-Hole stats for Grafana to InfluxDB
Stars: ✭ 126 (+240.54%)
Mutual labels:  grafana, grafana-dashboard
Cms Grafana Builder
helps you run a grafana server that include aliyun cms dashboard.
Stars: ✭ 26 (-29.73%)
Mutual labels:  grafana, grafana-dashboard
yearn-exporter
Realtime and historical Yearn metrics
Stars: ✭ 80 (+116.22%)
Mutual labels:  grafana, grafana-dashboard
Unifiedmetrics
Fully-featured metrics collection agent for Minecraft servers. Supports Prometheus and InfluxDB. Dashboard included out-of-box.
Stars: ✭ 29 (-21.62%)
Mutual labels:  grafana, grafana-dashboard
Legend
Legend builds and publishes Grafana dashboards for your services with prefilled metrics and alerts for your services.
Stars: ✭ 149 (+302.7%)
Mutual labels:  grafana, grafana-dashboard
Pihole Exporter
A Prometheus exporter for PI-Hole's Raspberry PI ad blocker
Stars: ✭ 352 (+851.35%)
Mutual labels:  grafana, grafana-dashboard
Grafana Dashboards
Grafana Dashboards
Stars: ✭ 228 (+516.22%)
Mutual labels:  grafana, grafana-dashboard
Dark
(grafana) Dashboards As Resources in Kubernetes
Stars: ✭ 190 (+413.51%)
Mutual labels:  grafana, grafana-dashboard
Darknet chinesetrading
🚇暗网中文网监控爬虫(DEEPMIX)
Stars: ✭ 649 (+1654.05%)
Mutual labels:  grafana, grafana-dashboard
gitana
Gitana is a lightweight dashboard sync
Stars: ✭ 61 (+64.86%)
Mutual labels:  grafana, grafana-dashboard
Reporter
Service that generates a PDF report from a Grafana dashboard
Stars: ✭ 581 (+1470.27%)
Mutual labels:  grafana, grafana-dashboard
Personal Influxdb
Import data from various APIs into InfluxDB
Stars: ✭ 51 (+37.84%)
Mutual labels:  grafana, grafana-dashboard
Version Checker
Kubernetes utility for exposing image versions in use, compared to latest available upstream, as metrics.
Stars: ✭ 371 (+902.7%)
Mutual labels:  grafana, grafana-dashboard
Grafana Influx Dashboard
Grafana InfluxDB scripted dashboard
Stars: ✭ 130 (+251.35%)
Mutual labels:  grafana, grafana-dashboard
Prometheus
A docker-compose stack for Prometheus monitoring
Stars: ✭ 3,383 (+9043.24%)
Mutual labels:  grafana, grafana-dashboard
Grabana
User-friendly Go library for building Grafana dashboards
Stars: ✭ 313 (+745.95%)
Mutual labels:  grafana, grafana-dashboard
Appmetrics
App Metrics is an open-source and cross-platform .NET library used to record and report metrics within an application.
Stars: ✭ 1,986 (+5267.57%)
Mutual labels:  grafana, grafana-dashboard
Pagerbeauty
📟✨ PagerDuty on-call widget for monitoring dashboard. Datadog and Grafana compatible
Stars: ✭ 250 (+575.68%)
Mutual labels:  grafana, grafana-dashboard

Monitoring Kafka with Burrow and Prometheus Operator

alt text

How to setup Prometheus Operator to scrape and store metrics exposed by burrow and jmx

FROM confluentinc/cp-kafka:4.1.0

COPY prometheus-exporter/jmx_prometheus_javaagent-0.3.0.jar /usr/share/java/kafka/jmx_prometheus_javaagent-0.3.0.jar
COPY prometheus-expoerter/jmx-exporter.yml /usr/share/java/kafka/jmx-exporter.yml

Kafka kubernetes manifests

Enable jmx-exporter java agent, add this options into env part of Kafka statefulset manifest:

        - name: KAFKA_OPTS
          value: >
            -javaagent:/usr/share/java/kafka/jmx_prometheus_javaagent-0.3.0.jar=5555:/usr/share/java/kafka/jmx-exporter.yml

Burrow

Burrow is recomended tool for monitoring consumer lag -- one of the most important metric for Kafka cluster. It has comprehensive documentation, please find it here: https://github.com/linkedin/Burrow/wiki
There is only one problem with Burrow, it doesn't expose metrics in Prometheus format, and this is the reason why we should use burrow-exporter docker container for converting metrics.

Create burrow docker image: We will mount config inside working container using Kubernetes configmap feature.

Create start.sh script and Dockerfile:

#!/bin/sh
KAFKA_VERSION=${KAFKA_VERSION:-0.10.1.0}
echo "start"
cat $CONFIG_FILE
exec $GOPATH/bin/Burrow -config-dir /etc/burrow/config/
FROM golang:alpine
RUN apk add --update bash curl git && apk add ca-certificates wget && update-ca-certificates && rm -rf /var/cache/apk/*
RUN go get github.com/linkedin/Burrow \
    github.com/golang/dep/cmd/dep
WORKDIR $GOPATH/src/github.com/linkedin/Burrow
RUN dep ensure && go install && mkdir -p /etc/burrow/
ADD ./ /etc/burrow/
WORKDIR /etc/burrow/
CMD ["/etc/burrow/start.sh"]

Prometheus Operator

Prometheus Operator uses k8s services and service monitors CRD for generating Prometheus config.

Apply this manifest inside namespace where Prometheus Operator works: Burrow Service, ConfigMap and Deployment:

kind: Service
apiVersion: v1
metadata:
  name: burrow
  labels:
    k8s-app: burrow
spec:
  selector:
    k8s-app: burrow
  ports:
  - name: metrics
    port: 8080
---
kind: ConfigMap
metadata:
  name: burrow-config
apiVersion: v1
data:
  burrow.toml: |-
    [general]
    access-control-allow-origin="*"
    
    [logging]
    level="info"
    
    [zookeeper]
    servers=["zk-0.zk-hs:2181","zk-1.zk-hs:2181","zk-2.zk-hs:2181", "zk-3.zk-hs:2181", "zk-4.zk-hs:2181"]
    
    [client-profile.kafka-profile]
    kafka-version="0.10.2.0"
    client-id="burrow-client"
    
    [cluster.gw]
    class-name="kafka"
    client-profile="kafka-profile"
    servers=["kafka-0.kafka-hs:9092","kafka-1.kafka-hs:9092","kafka-2.kafka-hs:9092","kafka-3.kafka-hs:9092","kafka-4.kafka-hs:9092"]
    topic-refresh=120
    offset-refresh=10
    
    [consumer.consumer_kafka]
    class-name="kafka"
    cluster="gw"
    servers=["kafka-0.kafka-hs:9092","kafka-1.kafka-hs:9092","kafka-2.kafka-hs:9092","kafka-3.kafka-hs:9092","kafka-4.kafka-hs:9092"]
    client-profile="kafka-profile"
    start-latest=true
    offsets-topic="__consumer_offsets"
    group-whitelist=".*"
    group-blacklist="^(console-consumer-|python-kafka-consumer-).*$"
    
    [httpserver.default]
    address=":8000"
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: burrow
  labels:
    k8s-app: burrow
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: burrow
  template:
    metadata:
      labels:
        k8s-app: burrow
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: burrow
        image: burrow
        ports:
        - name: api
          containerPort: 8000
        readinessProbe:
          httpGet:
            path: /burrow/admin
            port: 8000
        livenessProbe:
          httpGet:
            path: /burrow/admin
            port: 8000
        volumeMounts:
        - name: config
          mountPath: /etc/burrow/config
      - name: burrow-exporter
        image: burrow-exporter
        ports:
        - name: metrics
          containerPort: 8080
        env:
        - name: BURROW_ADDR
          value: http://localhost:8000
        - name: METRICS_ADDR
          value: 0.0.0.0:8080
        - name: INTERVAL
          value: "15"
      volumes:
      - name: config
        configMap:
          name: burrow-config

Modify Kafka service manifest -- add port on which jmx metrics exposed and add label for Prometheus Operator Kafka Service:

apiVersion: v1
kind: Service
metadata:
  name: kafka-hs
  labels:
    k8s-app: kafka-metrics
spec:
  clusterIP: None
  selector:
    app: kafka
  ports:
  - port: 9092
    name: internal
  - port: 9093
    name: client
  - port: 5555
    name: metrics

ServiceMonitor

Apply these manifests inside the namespace where your Prometheus Operator works, e.g. monitoring

Kafka ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kafka-metrics
  labels:
    k8s-app: kafka-metrics
spec:
  selector:
    matchLabels:
      k8s-app: kafka-metrics
  namespaceSelector:
    matchNames:
    - kafka-namespace
  endpoints:
  - port: metrics
    interval: 10s

Burrow ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: burrow
  labels:
    k8s-app: burrow
spec:
  selector:
    matchLabels:
      k8s-app: burrow
  namespaceSelector:
    matchNames:
    - monitoring
  endpoints:
  - port: metrics
    interval: 10s
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].