Apache Kafka® Confluent Platform on Kubernetes and OpenShift
This repo provides docker files, k8s manifests and OpenShift templates for Apache Kafka® Confluent Platform Community Edition. The reason why we're not using helm charts or the kafka operator at this time is because we like plain manifests which are more suitable for the learning phase.
Prerequisites
This implementation was developed and tested on EKS with k8s 1.10.3, with tk8 made k8s 1.11.2 and OpenShift 3.5 and 3.7.
If one would like to get it working on k8s < 1.9.x, the apiVersion for StatefulSets should be changed to apps/v1beta1 or apps/v1beta2 instead of apps/v1.
Quick start ephemeral
To deploy the ephemeral version for development create the kafka-confluent-5 namespace and deploy the kafka-broker, zookeeper and the related services as follow:
$ kubectl create ns kafka-confluent-5
$ kubectl create -f https://raw.githubusercontent.com/kubernauts/kafka-confluent-platform/master/k8s/streaming-ephemeral.yaml -n kafka-confluent-5$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-0 1/1 Running 0 1m
kafka-1 1/1 Running 0 1m
kafka-2 1/1 Running 0 1m
zoo-0 1/1 Running 0 1m
zoo-1 1/1 Running 0 1m
zoo-2 1/1 Running 0 1mQuick start persistent
To deploy the persistent version with PVCs, create the kafka-confluent-5 namespace, create the storage class gp2 (for aws-ebs here) and deploy the kafka-broker, zookeeper and the related services with persitent support as follow:
Please adapt the zone in the storage class where you've your k8s cluster:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2
annotations:
storageclass.kubernetes.io/is-default-class: "true"
labels:
kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: "us-east-1a"And run the follwoing commads:
$ kubectl create ns kafka-confluent-5
$ kubectl create -f https://raw.githubusercontent.com/kubernauts/kafka-confluent-platform/master/k8s/storageclass-aws-ebs.yaml
$ kubectl create -f https://raw.githubusercontent.com/kubernauts/kafka-confluent-platform/master/k8s/streaming-persistent-aws-ebs.yaml -n kafka-confluent-5Qucik start OpenShift
For OpenShift 3.5 create a project "kafka-confluent-5" in OpenShift and upload the template through OpenShift Webcosole:
Testing with Confluent Test Client
Deploy confluent test-client
kubectl create -f confluent-testclient-confluent-5.yamlCreate topics
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zoo --topic topic1 --create --partitions 30 --replication-factor 3List topics
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zookeeper:2181 --listDescribe topics
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zookeeper:2181 --describeProduce messages
kubectl exec -it confluent-client -- /usr/bin/kafka-console-producer --broker-list broker:9092 --topic topic1Consume messages
kubectl exec -it confluent-client -- /usr/bin/kafka-console-consumer --bootstrap-server broker:9092 --topic topic1 --from-beginningDelete topic
kubectl exec -it confluent-client -- /usr/bin/kafka-topics --zookeeper zookeeper:2181 --delete --topic topic1Testing with Kafkacat
Deploy Kafkacat:
kubectl create -f kafkacat.ymlExec into the producer container and write messages to topic1:
$ k exec -it kafkacat-12345 -c producer /bin/bash
$ for i in `seq 1 100`; do echo "hello kafka world" | kafkacat -b kafka:9092 -t topic1; doneIn another shell exec into the cosumer container and read messages:
$ k exec -it kafkacat-12345 -c consumer /bin/bash
# kafkacat -C -b kafka:9092 -t topic1Deploy Yahoo Kafka Manager with Helm
Make sure your k8s cluster is properly configured for Helm:
helm init
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'Clone the great implementation of Charles Martinot for the Yahoo Kafka Manager:
$ git clone https://github.com/MacTynow/kafka-manager-chart.git
$ cd kafka-manager-chart Adapt values.yaml
useKafkaZookeeper: true
zkHosts: "zoo"Install Yahoo Kafka Manager with Helm:
$ helm install .Follow the instructions of the helm output, and access the GUI, provide "zoo" as the Cluster Zookeeper Hosts (the headless service).
Rancher integration
Deploy Rancher on your k8s cluster with a single command:
kubectl create -f https://raw.githubusercontent.com/kubernauts/tk8/master/addons/rancher/master.yamlRancher will be up in 2 minutes, you can get the LB address through:
$ kubectl get svc -n cattle-systemAdd a project in Rancher by navigating to "Projects/Namespaces" in Rancher UI, e.g. "confluent-kafka-5-project" and move the namespace "confluent-kafka-5" namespace to the newly created project "confluent-kafka-5-project". With this you can add new users to the project and have control!
Monitoring and Alerting with Prometheus and Grafana through Rancher
Select your newly created project "confluent-kafka-5-project", navigate to Catalog Apps in the nice menu bar of Rancher and add Prometheus from Rancher's Library Catalog with the follwoing settings:
PROMETHEUS SERVER
Expose Prometheus using Layer 7 Load Balancer: false
Prometheus Service Type: ClusterIP
Prometheus Persistent Volume StorageClass: gp2 (whic was created through the storageclass on EKS)
Create Persistent Volume for Prometheus: trueGRAFANA SETTINGS
Expose Grafana using Layer 7 Load Balancer: false
Grafana Service Type: NodePort
Grafana Persistent Volume Enabled: true
Storage Class for Grafana: gp2ALERTMANAGER
Expose Alertmanager using Layer 7 Load Balancer: false
Alertmanager Service Type: ClusterIP
Create Persistent Volume for Alertmanager: true
Alertmanager Persistent Volume StorageClass: gp2And click on launch, after 2 minutes you shall get:
$ k get pods -n prometheus
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-6df98765f4-f79kh 2/2 Running 0 3m
prometheus-grafana-5bf7b6d949-pldn7 2/2 Running 0 3m
prometheus-kube-state-metrics-6584885ccf-l7tkm 1/1 Running 0 3m
prometheus-node-exporter-5qvjc 1/1 Running 0 3m
prometheus-node-exporter-bsp48 1/1 Running 0 3m
prometheus-node-exporter-f64cr 1/1 Running 0 3m
prometheus-node-exporter-r7t7c 1/1 Running 0 3m
prometheus-node-exporter-stdjg 1/1 Running 0 3m
prometheus-server-5959898967-s6hjd 2/2 Running 0 3mRun something like this to access the Grafana Dashboard:
$ kubectlforward prometheus-grafana-5bf7b6d949-pldn7 3000Import the kafka-overview_rev1.json and grafana-kafka-dashboard.json provided in the root of this repo as "Kafka Overview" Dashboard. You'll get a bunch of other very useful dashboard provided throught the Rancher installation.
Clean up
To clean up your environment, run:
$ k delete all -l app=kafka -n kafka-confluent-5
$ k delete pvc --all
$ k delete ns kafka-confluent-5Screenshots
Useful links
https://github.com/edenhill/kafkacat
https://github.com/Yolean/kubernetes-kafka
https://docs.confluent.io/current/app-development/kafkacat-usage.html
https://hackernoon.com/a-blockchain-experiment-with-apache-kafka-97ee0ab6aefc
https://hackernoon.com/simple-chatops-with-kafka-grafana-prometheus-and-slack-764ece59e707
Get in touch
Join us on Kubernauts Slack Channel: