KafkaMonitor
the design and implementation reference to kt and kafkatools
Install
go get github.com/smallnest/kafka_monitor
Run
# change the below options with your environments
[email protected]
smtpHost=xxxxxx
smtpPort=25
[email protected]
smtpPassword=xxxxxx
brokers=10.77.11.1:9092,10.77.11.2:9092,10.77.11.3:9092,10.77.11.4:9092
version=0.9.0.1
zkAddr=10.77.0.1:2181,10.77.0.2:2181,10.77.0.3:2181,10.77.0.4:2181,10.77.0.5:2181
./kafka_monitor -brokers ${brokers} -version ${version} -email=${informEmail} -smtp.host ${smtpHost} -smtp.port ${smtpPort) -smtp.user ${smtpUser} -smtp.password ${smtpPassword} -zkAddr ${zkAddr} -basePath "/kafka"
Output
Once some partitions are not writable, down or some consumers don't work, an email will be sent to your email with details.
Normal kafka cluster info is output in console:
Brokers: 10.77.11.1:9092,10.77.11.2:9092,10.77.11.4:9092:10.77.11.3:9092
Version: 0.9.0.1
Topic: products, Group: default, Partitions: 32
+-----------+-------------------+--------+---------------+---------------+
| PARTITION | LEADER ADDRESS | LEADER | REPLICAS | ISR |
+-----------+-------------------+--------+---------------+---------------+
| 0 | 10.77.11.1:9092 | 1 | [1 2 3] | [1 2 3] |
| 1 | 10.77.11.2:9092 | 2 | [1 2 4] | [1 2 4] |
| 2 | 10.77.11.4:9092 | 4 | [2 4 3] | [2 4 3] |
| 3 | 10.77.11.3:9092 | 3 | [1 4 3] | [1 4 3] |
| 4 | 10.77.11.1:9092 | 1 | [1 2 4] | [1 2 4] |
| 5 | 10.77.11.2:9092 | 2 | [2 4 3] | [2 4 3] |
| 6 | 10.77.11.4:9092 | 4 | [1 4 3] | [1 4 3] |
| 7 | 10.77.11.3:9092 | 3 | [1 2 3] | [1 2 3] |
| 8 | 10.77.11.1:9092 | 1 | [1 4 3] | [1 4 3] |
| 9 | 10.77.11.2:9092 | 2 | [1 2 3] | [1 2 3] |
| 10 | 10.77.11.4:9092 | 4 | [1 2 4] | [1 2 4] |
| 11 | 10.77.11.3:9092 | 3 | [2 4 3] | [2 4 3] |
| 12 | 10.77.11.1:9092 | 1 | [1 2 3] | [1 2 3] |
| 13 | 10.77.11.2:9092 | 2 | [1 2 4] | [1 2 4] |
| 14 | 10.77.11.4:9092 | 4 | [2 4 3] | [2 4 3] |
| 15 | 10.77.11.3:9092 | 3 | [1 4 3] | [1 4 3] |
| 16 | 10.77.11.1:9092 | 1 | [1 2 4] | [1 2 4] |
| 17 | 10.77.11.2:9092 | 2 | [2 4 3] | [2 4 3] |
| 18 | 10.77.11.4:9092 | 4 | [1 4 3] | [1 4 3] |
| 19 | 10.77.11.3:9092 | 3 | [1 2 3] | [1 2 3] |
| 20 | 10.77.11.1:9092 | 1 | [1 4 3] | [1 4 3] |
| 21 | 10.77.11.2:9092 | 2 | [1 2 3] | [1 2 3] |
| 22 | 10.77.11.4:9092 | 4 | [1 2 4] | [1 2 4] |
| 23 | 10.77.11.3:9092 | 3 | [2 4 3] | [2 4 3] |
| 24 | 10.77.11.1:9092 | 1 | [1 2 3] | [1 2 3] |
| 25 | 10.77.11.2:9092 | 2 | [1 2 4] | [1 2 4] |
| 26 | 10.77.11.4:9092 | 4 | [2 4 3] | [2 4 3] |
| 27 | 10.77.11.3:9092 | 3 | [1 4 3] | [1 4 3] |
| 28 | 10.77.11.1:9092 | 1 | [1 2 4] | [1 2 4] |
| 29 | 10.77.11.2:9092 | 2 | [2 4 3] | [2 4 3] |
| 30 | 10.77.11.4:9092 | 4 | [1 4 3] | [1 4 3] |
| 31 | 10.77.11.3:9092 | 3 | [1 2 3] | [1 2 3] |
+-----------+-------------------+--------+---------------+---------------+
+-----------+------------+--------------+-----+
| PARTITION | END OF LOG | GROUP OFFSET | LAG |
+-----------+------------+--------------+-----+
| 0 | 1106509557 | 1106509542 | 15 |
| 1 | 1106517793 | 1106517779 | 14 |
| 2 | 1106573067 | 1106573047 | 20 |
| 3 | 1106531253 | 1106531253 | 0 |
| 4 | 1106529394 | 1106529385 | 9 |
| 5 | 11065504 | 1106550146 | 4 |
| 6 | 1106467253 | 1106467248 | 5 |
| 7 | 1106479964 | 1106479964 | 0 |
| 8 | 1106464607 | 1106464594 | 13 |
| 9 | 1106503601 | 1106503589 | 12 |
| 10 | 1106543568 | 1106543552 | 16 |
| 11 | 1106558780 | 1106558768 | 12 |
| 12 | 1106494251 | 1106494238 | 13 |
| 13 | 1106561496 | 1106561490 | 6 |
| 14 | 1106521678 | 1106521673 | 5 |
| 15 | 1106569720 | 1106569686 | 34 |
| 16 | 1106573067 | 1106573049 | 18 |
| 17 | 1106521317 | 1106521305 | 12 |
| 18 | 1106534494 | 1106534484 | 10 |
| 19 | 1106481325 | 1106481268 | 57 |
| 20 | 1106530486 | 1106530465 | 21 |
| 21 | 1106500650 | 1106500624 | 26 |
| 22 | 1106552135 | 1106552113 | 22 |
| 23 | 1106551799 | 1106551771 | 28 |
| 24 | 1106564761 | 1106564750 | 11 |
| 25 | 1106537078 | 1106537064 | 14 |
| 26 | 1106573795 | 1106573781 | 14 |
| 27 | 1106493569 | 1106493554 | 15 |
| 28 | 1106500029 | 1106500015 | 14 |
| 29 | 1106528095 | 1106528078 | 17 |
| 30 | 1106493651 | 1106493629 | 22 |
| 31 | 1106562315 | 1106562312 | 3 |
+-----------+------------+--------------+-----+
| TOTAL | 482 |
+-----------+------------+--------------+-----+