GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → sky-big → pulsar-operator

sky-big / pulsar-operator

Licence: MIT license
Pulsar Operator creates/configures/manages pulsar clusters atop Kubernetes

Programming Languages

go
31211 projects - #10 most used programming language
shell
77523 projects
Makefile
30231 projects
Dockerfile
14818 projects

Labels

kubernetespulsarkubernetes-operatoroperator-sdkpulsar-operator

Projects that are alternatives of or similar to pulsar-operator

pulp-operator
Kubernetes Operator for Pulp 3. Under active development.
Stars: ✭ 32 (-52.94%)
Mutual labels:  kubernetes-operator, operator-sdk
rabbitmq-operator
RabbitMQ Kubernetes operator
Stars: ✭ 16 (-76.47%)
Mutual labels:  kubernetes-operator, operator-sdk
netris-operator
The Kubernetes Operator for Netris
Stars: ✭ 28 (-58.82%)
Mutual labels:  kubernetes-operator
astarte-kubernetes-operator
Astarte Kubernetes Operator
Stars: ✭ 18 (-73.53%)
Mutual labels:  kubernetes-operator
infinispan-operator
Infinispan Operator
Stars: ✭ 32 (-52.94%)
Mutual labels:  kubernetes-operator
pulsar-io-kafka
Pulsar IO Kafka Connector
Stars: ✭ 24 (-64.71%)
Mutual labels:  pulsar
kubexray
JFrog KubeXray scanner on Kubernetes
Stars: ✭ 22 (-67.65%)
Mutual labels:  kubernetes-operator
tgip
TGIP (TGI Pulsar) is a weekly live video streaming about Apache Pulsar and its ecosystem.
Stars: ✭ 17 (-75%)
Mutual labels:  pulsar
varnish-operator
Run and manage Varnish clusters on Kubernetes
Stars: ✭ 47 (-30.88%)
Mutual labels:  kubernetes-operator
aws-nlb-helper-operator
Simple operator to manage AWS NLB attributes using Kubernetes Service object annotations
Stars: ✭ 23 (-66.18%)
Mutual labels:  operator-sdk
kubectlsafe
Safe operations in kubectl with plugin kubectlsafe
Stars: ✭ 36 (-47.06%)
Mutual labels:  kubernetes-operator
freya
Scala Kubernetes Operator library
Stars: ✭ 40 (-41.18%)
Mutual labels:  kubernetes-operator
stackgres
StackGres Operator, Full Stack PostgreSQL on Kubernetes // !! Mirror repository of https://gitlab.com/ongresinc/stackgres, only accept Merge Requests there.
Stars: ✭ 479 (+604.41%)
Mutual labels:  kubernetes-operator
cert-operator
An OpenShift controller using the Operator SDK for managing TLS certficate lifecycle
Stars: ✭ 27 (-60.29%)
Mutual labels:  operator-sdk
cmak-operator
CMAK (prev. Kafka Manager) for Kubernetes
Stars: ✭ 45 (-33.82%)
Mutual labels:  kubernetes-operator
kubeflare
A Kubernetes Operator to manage Cloudflare settings via a declarative Kubernetes API
Stars: ✭ 50 (-26.47%)
Mutual labels:  kubernetes-operator
pulsar-core
🚀 Handy dynamic styles utilities for React Native and React Native Web.
Stars: ✭ 27 (-60.29%)
Mutual labels:  pulsar
sieve
Automatic Reliability Testing for Kubernetes Controllers
Stars: ✭ 183 (+169.12%)
Mutual labels:  kubernetes-operator
kotary
Managing Kubernetes Quota with confidence
Stars: ✭ 85 (+25%)
Mutual labels:  kubernetes-operator
pulsar-flex
Pulsar Flex is a modern Apache Pulsar client for Node.js, developed to be independent of C++.
Stars: ✭ 43 (-36.76%)
Mutual labels:  pulsar
View All Similar Projects ➔

Pulsar Operator

Overview

Pulsar Operator is to manage Pulsar service instances deployed on the Kubernetes cluster. It is built using the Operator SDK, which is part of the Operator Framework.

Quick Start

Deploy Pulsar Operator

  1. Clone the project on your Kubernetes cluster master node:
$ git clone https://github.com/sky-big/pulsar-operator.git
$ cd pulsar-operator
  1. To deploy the Pulsar Operator on your Kubernetes cluster, please run the following script:
$ make install
  1. Use command kubectl get pods to check Pulsar Operator deploy status like:
$ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
pulsar-operator-564b5d75d-jllzk           1/1     Running   0          108s

Now you can use the CRDs provide by Pulsar Operator to deploy your Pulsar Cluster.

Define Your Pulsar Cluster

  1. Check the file pulsar_v1alpha1_pulsarcluster_cr.yamlin the deploy/crd directory, for example:
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  zookeeper:
    size: 3
  autoRecovery:
    size: 3
  bookie:
    size: 3
  broker:
    size: 3
  proxy:
    size: 3

which defines pulsar cluster zookeeper, bookkeeper, broker, proxy components configuration

  1. If you need pulsar prometheus, grafana need configuration, for example:
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  zookeeper:
    size: 3
  autoRecovery:
    size: 3
  bookie:
    size: 3
  broker:
    size: 3
  proxy:
    size: 3
  monitor:
    enable: true                // true/false: active monitor
    prometheusPort: 30002       // prometheus expose port on kubernetes
    grafanaPort: 30003          // grafana expose port on kubernetes
    ingress:                    // ingress configuration
      enable: true
      annotations:
        kubernetes.io/ingress.class: "nginx"
  1. If you need pulsar manager expose by ingress, for example:
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  zookeeper:
    size: 3
  autoRecovery:
    size: 3
  bookie:
    size: 3
  broker:
    size: 3
  proxy:
    size: 3
  manager:
    enable: true                            // true/false: active manager
    host: manager.pulsar.com
    annotations:
      kubernetes.io/ingress.class: "nginx"

Create Your Pulsar Cluster

  1. Deploy the pulsar cluster by running:
$ kubectl create -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml
  1. Use command kubectl get podsto check example Pulsar Cluster status like:
$ kubectl get pods
NAME                                                              READY   STATUS      RESTARTS   AGE
example-pulsarcluster-bookie-autorecovery-deployment-6775f2sbxj   1/1     Running     0          67s
example-pulsarcluster-bookie-autorecovery-deployment-6775fdqhmb   1/1     Running     0          67s
example-pulsarcluster-bookie-autorecovery-deployment-6775ftnftd   1/1     Running     0          67s
example-pulsarcluster-bookie-statefulset-0                        1/1     Running     0          68s
example-pulsarcluster-bookie-statefulset-1                        1/1     Running     0          55s
example-pulsarcluster-bookie-statefulset-2                        1/1     Running     0          42s
example-pulsarcluster-broker-deployment-5bb58577b4-4tr4l          1/1     Running     0          67s
example-pulsarcluster-broker-deployment-5bb58577b4-6vzhm          1/1     Running     0          67s
example-pulsarcluster-broker-deployment-5bb58577b4-mzphh          1/1     Running     0          67s
example-pulsarcluster-init-cluster-metadata-job-98rsd             0/1     Completed   0          80s
example-pulsarcluster-proxy-deployment-6555968487-7df5l           1/1     Running     0          67s
example-pulsarcluster-proxy-deployment-6555968487-cfxl7           1/1     Running     0          67s
example-pulsarcluster-proxy-deployment-6555968487-cxhc6           1/1     Running     0          67s
example-pulsarcluster-zookeeper-statefulset-0                     1/1     Running     0          2m2s
example-pulsarcluster-zookeeper-statefulset-1                     1/1     Running     0          109s
example-pulsarcluster-zookeeper-statefulset-2                     1/1     Running     0          95s

Storage of Pulsar Cluster

  1. Use EmptyDir Volume For Test(Default)

  2. Use PVC For Storage

apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  zookeeper:
    size: 3
  bookie:
    size: 3
    storageClassName: "fast"                // storage class name
    journalStorageCapacity: 10              // bookie journal volume capacity(Gi)
    ledgersStorageCapacity: 10              // bookie ledgers volume capacity(Gi)
  broker:
    size: 3
  proxy:
    size: 3
    
$ kubectl get pvc
NAME                                                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
journal-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-0   Bound    pvc-8c7247e0-9a1f-4eba-9a1f-8ad10847d54d   10Gi       RWO            fast           2m
journal-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-1   Bound    pvc-27e5b83b-6ea2-47e2-a64e-783772506959   10Gi       RWO            fast           84s
journal-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-2   Bound    pvc-88ccca2e-0f6b-4ae9-b5c6-1b7bb57bdc69   10Gi       RWO            fast           62s
ledgers-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-0   Bound    pvc-3afc1103-7992-41e1-9aa5-762681ac5e5d   10Gi       RWO            fast           2m
ledgers-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-1   Bound    pvc-dd848627-6838-4cda-be77-83fe5b0b7c53   10Gi       RWO            fast           84s
ledgers-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-2   Bound    pvc-98aea962-1a7a-4c4c-a058-cef9bccacb6c   10Gi       RWO            fast           62s

Horizontal Scale Pulsar Cluster

Scale Pulsar Proxy

  1. If you want to enlarge your proxy component. Modify your CR file pulsar_v1alpha1_pulsarcluster_cr.yaml, increase the field size to the number you want, for example, from size: 3 to size: 5
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  proxy:
    size: 5
  1. After configuring the size fields, simply run:
$ kubectl apply -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml

Scale Pulsar Broker

  1. If you want to enlarge your broker component. Modify your CR file pulsar_v1alpha1_pulsarcluster_cr.yaml, increase the field size to the number you want, for example, from size: 3 to size: 5
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  broker:
    size: 5
  1. After configuring the size fields, simply run:
$ kubectl apply -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml

Scale Pulsar Bookie

  1. If you want to enlarge your bookie component. Modify your CR file pulsar_v1alpha1_pulsarcluster_cr.yaml, increase the field size to the number you want, for example, from size: 3 to size: 5
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  bookie:
    size: 5
  1. After configuring the size fields, simply run:
$ kubectl apply -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml

Local Test On Kubernetes

Start Local Test

  1. Install kubernetes cluster

  2. Install golang on kubernetes master node

  3. Install operator sdkInstall Operator SDK on kubernetes master node

  4. Clone project to the kubernetes master node(project must in GOPATH):

$ git clone https://github.com/sky-big/pulsar-operator.git
$ cd pulsar-operator
  1. Execute script on kubernetes master node:
$ make start-local

Stop Local Test

  1. Execute script on kubernetes master node:
$ make stop-local

Compile Pulsar Operator

$ make build

Build Pulsar Operator Image

$ make image

Generate CRD Code And Project Vendor Code

$ make generate

Discussions Happen Here

Slack

(1). join

(2). pulsar-operator-dev

(3). pulsar-operator-user

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].