GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → nbogojevic → kubernetes-kafka

nbogojevic / kubernetes-kafka

Licence: other
Managing kafka clusters in Kubernetes and OpenShift

Programming Languages

shell
77523 projects

Labels

kuberneteskafkaopenshiftkafka-container

Projects that are alternatives of or similar to kubernetes-kafka

Selinon
An advanced distributed task flow management on top of Celery
Stars: ✭ 237 (+1216.67%)
Mutual labels:  openshift
kubernetes.core
The collection includes a variety of Ansible content to help automate the management of applications in Kubernetes and OpenShift clusters, as well as the provisioning and maintenance of clusters themselves.
Stars: ✭ 159 (+783.33%)
Mutual labels:  openshift
ocp4upc
OCP4 Upgrade Paths Checker
Stars: ✭ 30 (+66.67%)
Mutual labels:  openshift
openshift-starter-guides
Getting Started with OpenShift for Developers Workshop
Stars: ✭ 35 (+94.44%)
Mutual labels:  openshift
openshift4-upi-homelab
OpenShift 4 User Provisioned Infrastructure Homelab
Stars: ✭ 15 (-16.67%)
Mutual labels:  openshift
anthill
A Kubernetes/OpenShift operator to manage Gluster clusters
Stars: ✭ 35 (+94.44%)
Mutual labels:  openshift
Openshift Cartridge Nodejs
Custom cartridge for OpenShift providing the lastest version of Node.js.
Stars: ✭ 227 (+1161.11%)
Mutual labels:  openshift
apb-examples
A repository of example ansible-playbook bundles. THIS REPO IS DEPRECATED. Please look at https://github.com/ansibleplaybookbundle/ For updated examples.
Stars: ✭ 15 (-16.67%)
Mutual labels:  openshift
openshift-cartridge-mysql
Custom cartridge for OpenShift providing MySQL 5.7.17
Stars: ✭ 13 (-27.78%)
Mutual labels:  openshift
camunda-cloud-helm
Camunda Platform 8 Self-Managed Helm charts
Stars: ✭ 41 (+127.78%)
Mutual labels:  openshift
openshift-json-schema
A set of JSON schemas for various OpenShift versions, extracted from the OpenAPI definitions
Stars: ✭ 23 (+27.78%)
Mutual labels:  openshift
pulp-operator
Kubernetes Operator for Pulp 3. Under active development.
Stars: ✭ 32 (+77.78%)
Mutual labels:  openshift
ChRIS ultron backEnd
Backend for ChRIS
Stars: ✭ 28 (+55.56%)
Mutual labels:  openshift
Openshift Examples
Openshift Examples - This repo does not provide end to end example but rather act as a rough draft for my work. use with caution. Buzzme at @twitter
Stars: ✭ 247 (+1272.22%)
Mutual labels:  openshift
Chowkidar
A kubernetes controller that watches/observes events & then takes configured actions – [✩Star] if you're using it!
Stars: ✭ 55 (+205.56%)
Mutual labels:  openshift
Openwhisk Deploy Kube
The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
Stars: ✭ 231 (+1183.33%)
Mutual labels:  openshift
jetlag
Automation to deploy Bare-metal OpenShift leveraging the Assisted-Installer
Stars: ✭ 18 (+0%)
Mutual labels:  openshift
dockerevil
My security researches involving Docker and Openshift
Stars: ✭ 28 (+55.56%)
Mutual labels:  openshift
s2i-ruby-container
Ruby container images based on Red Hat Software Collections and intended for OpenShift and general usage, that provide a platform for building and running Ruby applications. Users can choose between Red Hat Enterprise Linux, Fedora, and CentOS based images.
Stars: ✭ 55 (+205.56%)
Mutual labels:  openshift
backup-repository
Backup storage for E2E GPG-encrypted files, with multi-user, quotas, versioning, using a object storage (S3/Min.io/GCS etc.) and deployed on Kubernetes or standalone.
Stars: ✭ 21 (+16.67%)
Mutual labels:  openshift
View All Similar Projects ➔

Managing Kafka clusters in Kubernetes

Installing

The installation is done using helm chart:

helm install kafka-cluster

The chart will install zookeeper and kafka clusters as StatefulSets, configure needed services and monitoring.

Running in OpenShift

Installation in OpenShift can be done using provided openshift templates. For details see openshift/README.md

Helm Chart Parameters

kafka:
  ## kafka.imageRegistry is the address of the container image registry to use
  # imageRegistry: docker.io
  ## kafka.pullPolicy is the kubernetes policy to use for pulling images: Always, IfNotPresent, Never
  pullPolicy: IfNotPresent
  ## kafka.image is the name of kafka container image to use
  image: nbogojevic/kafka
  ## kafka.tag is the tag of kafka container image to use
  tag: latest
  ## kafka.serviceName is the service name used by clients to find the cluster
  serviceName: kafka
  ## kafka.dnsName is the name used by StatefulSet to identify instances
  dnsName: broker
  ## kafka.nodes is set to number of nodes in kafka cluster
  nodes: 3
  ## kafka.jvmFlags are default java options to use with kafka
  jvmFlags: -Xmx1G
  ## kafka.storageSize is the default size of storage when using persistent volumes
  storageSize: 2GiB
  ## kafka.persistent is set to true to use persitent volumes
  persistent: false
  ## set to true if clients must authenticate. Only supports plaintext SASL mechanism.
  secured: true
  ## set to true if communication with kafka should be encrypted. TODO certificate management
  # tls: false
  # Set to true if topics can be deleted.
  deleteTopicEnable: false
  # Then name of the admin user. If not specified a random name is generated.
  #adminUser: admin
  # Then name of the admin password. If not specified a random password is generated.
  #adminPassword: secret-password
  # Then name of the client user. If not specified a random name is generated.
  #clientUser: client
  # Then name of the client password. If not specified a random password is generated.
  #clientPassword: secret-password
  zookeeperService: zookeeper
  # Default timeout in ms when connecting to zookeeper
  zookeperTimeout: 6000
  ## kafka.nodeSelector can be set with node labels to use when choosing nodes to deploy
  nodeSelector:

zookeeper:
  ## zookeeper.imageRegistry is the address of the container image registry to use
  # imageRegistry: docker.io
  ## kafka.pullPolicy is the kubernetes policy to use for pulling images: Always, IfNotPresent, Never
  pullPolicy: IfNotPresent
  ## zookeeper.image is the name of zookeeper container image to use
  image: nbogojevic/zookeeper
  ## zookeeper.tag is the tag of zookeeper container image to use
  tag: latest
  ## zookeeper.serviceName is the service name used by clients to find the cluster
  serviceName: zookeeper
  ## zookeeper.dnsName is the name used by StatefulSet to identify instances
  dnsName: zoo
  ## zookeeper.nodes is set to number of nodes in zookeeper cluster
  nodes: 3 
  ## zookeeper.jvmFlags are default java options to use with kafka
  jvmFlags: -Xmx512m
  ## zookeeper.nodeSelector can be set with node labels to use when choosing nodes to deploy
  nodeSelector:     

global:
  ## applicationName is the default name of the application
  applicationName: kafka-cluster

  ## hawkular is set to true to enable hawkular metrics
  hawkular: false   

  ## prometheus is set to true to enable prometheus metrics
  prometheus: true

  ## jolokia is set to true if jolokia JMX endpoint should be exposed
  jolokia: true

  ## jolokiaUsername is set to the name of jolokia user for basic authentication
  # jolokiaUsername: jolokia

  ## jolokiaPassword can be set to the password to use. If not specified, a random password is generated
  # jolokiaPassword: jolokia

Kafka

Kafka as Kubernetes StatefulSet

Kafka runs as a Kubernetes StatefulSet (or PetSet in older versions - helm chart automatically chooses the supported resource type).

Storage

Kafka can be run with or without persitent volumes.

When running with persitent volumes, the data is saved on the instance or the node crash. When running without persitent volumes, the data is stored locally on the node a kubernetes node loss may result in data loss, so high-availability and resiliency should be supported by kafka mechanisms, e.g. setting replication factor sufficiently high.

As for best performances, one might perfer specific local storage characteristics (SSD), helm chart allows usage of node selector to select instances where kafka would be deployed.

Kafka container image

Default kafka container image is based on fabric8/s2i-java image. The docker build is located in docker-kafka/.

Zookeeper

Zookeeper also runs as a Stateful Set.

Zookeeper uses local storage, so if you lose your zookeeper cluster, kafka will be unaware that persisted topics exist. The topic data is still present on kafka brokers, but you will need to re-create topics. This can be done using kafka-operator.

Zookeeper container image

Default zookeeper container image is based on fabric8/s2i-java image. The docker build is located in docker-zookeeper/.

Monitoring

The startup script in docker images will detect if prometheus jmx_exporter, jolokia or agant-bond are installed and enable them. Each image is pre-configured to expose relevant JMX MBeans via prometheus jmx_exporter.

If prometheus is enabled via helm variables, StatefulSet pods will be annotated with standard prometheus scraping annotations.

Building

Building docker images:

docker build docker-kafka -t nbogojevic/kafka -f docker-kafka/Dockerfile
docker build docker-zookeeper -t nbogojevic/zookeeper -f docker-zookeeper/Dockerfile

Checking helm chart:

helm init
helm lint kafka-cluster
helm install kafka-cluster
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].