sundy-li / Burrowx
Licence: apache-2.0
Kafka consumer lag monitor
Stars: ✭ 50
Programming Languages
go
31211 projects - #10 most used programming language
Projects that are alternatives of or similar to Burrowx
Real Time Stream Processing Engine
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
Stars: ✭ 37 (-26%)
Mutual labels: kafka
Akka.streams.kafka
Kafka Akka.Streams connectors - part of the Alpakka project
Stars: ✭ 42 (-16%)
Mutual labels: kafka
Rssa
RSS-Anything, get updates about anything you can reach with an url. Like RSS, but for anything.
Stars: ✭ 46 (-8%)
Mutual labels: monitor
Connection Pool Client
💥 A simple multi-purpose connection pool client (Kafka & Hbase & Redis & RMDB & Socket & Http)
Stars: ✭ 40 (-20%)
Mutual labels: kafka
Kafka Workshop
Materials (slides and code) for Kafka and Kafka Streams Workshop
Stars: ✭ 44 (-12%)
Mutual labels: kafka
Anotherkafkamonitor Akm
Another app which used to monitor the progress of Kafka Producer and Consumer
Stars: ✭ 36 (-28%)
Mutual labels: kafka
Spark streaming kafka offset
SparkStreaming中利用MySQL保存Kafka偏移量保证0数据丢失
Stars: ✭ 41 (-18%)
Mutual labels: kafka
Go Kafka Avro
A library provides consumer/producer to work with kafka, avro and schema registry
Stars: ✭ 39 (-22%)
Mutual labels: kafka
Storm Dynamic Spout
A framework for building spouts for Apache Storm and a Kafka based spout for dynamically skipping messages to be processed later.
Stars: ✭ 40 (-20%)
Mutual labels: kafka
Kowl
Apache Kafka Web UI for exploring messages, consumers, configurations and more with a focus on a good UI & UX.
Stars: ✭ 1,036 (+1972%)
Mutual labels: kafka
Delta Architecture
Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline
Stars: ✭ 43 (-14%)
Mutual labels: kafka
Rafka
Kafka proxy with a simple API, speaking the Redis protocol
Stars: ✭ 49 (-2%)
Mutual labels: kafka
Awesome Recommendation Engine
The purpose of this tiny project is to put things together with the know how that i learned from the course big data expert from formacionhadoop.com The idea is to show how to play with apache spark streaming, kafka,mongo, spark machine learning algorithms.
Stars: ✭ 47 (-6%)
Mutual labels: kafka
Examples
Demo applications and code examples for Confluent Platform and Apache Kafka
Stars: ✭ 571 (+1042%)
Mutual labels: kafka
https://github.com/danielqsj/kafka_exporter
WARNING: This repo is not maintained anymore, please useburrowx - kafka offset lag monitor,stored by influxdb
A simple, lightweight kafka offset monitor, currently metrics stored by influxdb. Motivated by Burrow, but much faster and cleaner and more stable. burrowx is good integration with influxdb and grafana.
DemoView
Install
-
By release binary
Just click the release links
-
By source build
$ go get github.com/sundy-li/burrowx
$ cd $GOPATH/src/github.com/sundy-li/burrowx
$ go build && go install
Running burrowx
## new workspace for burrowx
mkdir -p /data/app/burrowx
## cd to the workspace
cd /data/app/burrowx
## cp the files to here
cp $GOPATH/bin/burrowx ./
cp -rf $GOPATH/src/github.com/sundy-li/burrowx/config ./
## you should create the burrowx database in influxdb manually
## then modify server.json file config and run it
./burrowx
Docker
A Docker file is available which builds this project on top of an Alpine Linux image.
- Create your desired configuration files (server.yaml, logging.xml)
- Run
docker build -t burrowx .
It will include the server.yaml and logging.xml and automatically start service. - Run the container on your favourite container platform
Test the data
-
Create a new test topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 8 --topic test_burrowx_topic
-
Produce the data to the topic
for i in `seq 1 10000`;do echo "33" | bin/kafka-console-producer.sh --topic test_burrowx_topic --broker-list localhost:9092 ; sleep 1; done
-
Create a consumer to consume the data
$ pip install kafka
from kafka import KafkaConsumer consumer = KafkaConsumer('test_burrowx_topic', group_id='my_group2') for msg in consumer: print(msg) print("end")
Then you will find the data in the influxdb database burrowx
.
Schema in influxdb
-
cluster
: cluster name -
topic
: topic name -
consumer_group
: group name -
partition
: partition id -
logsize
: partition logsize -
offsize
: partition consumer offsize -
lag
: partition consumer log
Query Example
SELECT sum("logsize") FROM "consumer_metrics" WHERE ("cluster" = 'your_cluster' AND "topic" = 'your_topic' AND "consumer_group" = 'your_consumer') AND $timeFilter GROUP BY time(10s)
SELECT sum("offsize") FROM "consumer_metrics" WHERE ("cluster" = 'your_cluster' AND "topic" = 'your_topic' AND "consumer_group" = 'your_consumer') AND $timeFilter GROUP BY time(10s)
SELECT sum("lag") FROM "consumer_metrics" WHERE ("cluster" = 'your_cluster' AND "topic" = 'your_topic' AND "consumer_group" = 'your_consumer') AND $timeFilter GROUP BY time(10s)
Features
- Light weight and extremely simple to use, metrics are stored in influxdb, and could be easily viewed on grafana
- Only support kafka version >= 0.9.X, which stores the consumer offsets in the topic
__consumer_offsets
,if you are using kafka 0.8.X, try my previous repohttps://github.com/shunfei/Dcmonitor
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].