Pulsar Operator creates/configures/manages pulsar clusters atop Kubernetes
Pulsar Operator is to manage Pulsar service instances deployed on the Kubernetes cluster. It is built using the Operator SDK, which is part of the Operator Framework.
$ git clone https://github.com/sky-big/pulsar-operator.git
$ cd pulsar-operator
$ make install
kubectl get pods
to check Pulsar Operator deploy status like:$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pulsar-operator-564b5d75d-jllzk 1/1 Running 0 108s
Now you can use the CRDs provide by Pulsar Operator to deploy your Pulsar Cluster.
pulsar_v1alpha1_pulsarcluster_cr.yaml
in the deploy/crd directory, for example:apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
zookeeper:
size: 3
autoRecovery:
size: 3
bookie:
size: 3
broker:
size: 3
proxy:
size: 3
which defines pulsar cluster zookeeper, bookkeeper, broker, proxy components configuration
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
zookeeper:
size: 3
autoRecovery:
size: 3
bookie:
size: 3
broker:
size: 3
proxy:
size: 3
monitor:
enable: true // true/false: active monitor
prometheusPort: 30002 // prometheus expose port on kubernetes
grafanaPort: 30003 // grafana expose port on kubernetes
ingress: // ingress configuration
enable: true
annotations:
kubernetes.io/ingress.class: "nginx"
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
zookeeper:
size: 3
autoRecovery:
size: 3
bookie:
size: 3
broker:
size: 3
proxy:
size: 3
manager:
enable: true // true/false: active manager
host: manager.pulsar.com
annotations:
kubernetes.io/ingress.class: "nginx"
$ kubectl create -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml
kubectl get pods
to check example Pulsar Cluster status like:$ kubectl get pods
NAME READY STATUS RESTARTS AGE
example-pulsarcluster-bookie-autorecovery-deployment-6775f2sbxj 1/1 Running 0 67s
example-pulsarcluster-bookie-autorecovery-deployment-6775fdqhmb 1/1 Running 0 67s
example-pulsarcluster-bookie-autorecovery-deployment-6775ftnftd 1/1 Running 0 67s
example-pulsarcluster-bookie-statefulset-0 1/1 Running 0 68s
example-pulsarcluster-bookie-statefulset-1 1/1 Running 0 55s
example-pulsarcluster-bookie-statefulset-2 1/1 Running 0 42s
example-pulsarcluster-broker-deployment-5bb58577b4-4tr4l 1/1 Running 0 67s
example-pulsarcluster-broker-deployment-5bb58577b4-6vzhm 1/1 Running 0 67s
example-pulsarcluster-broker-deployment-5bb58577b4-mzphh 1/1 Running 0 67s
example-pulsarcluster-init-cluster-metadata-job-98rsd 0/1 Completed 0 80s
example-pulsarcluster-proxy-deployment-6555968487-7df5l 1/1 Running 0 67s
example-pulsarcluster-proxy-deployment-6555968487-cfxl7 1/1 Running 0 67s
example-pulsarcluster-proxy-deployment-6555968487-cxhc6 1/1 Running 0 67s
example-pulsarcluster-zookeeper-statefulset-0 1/1 Running 0 2m2s
example-pulsarcluster-zookeeper-statefulset-1 1/1 Running 0 109s
example-pulsarcluster-zookeeper-statefulset-2 1/1 Running 0 95s
Use EmptyDir Volume For Test(Default)
Use PVC For Storage
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
zookeeper:
size: 3
bookie:
size: 3
storageClassName: "fast" // storage class name
journalStorageCapacity: 10 // bookie journal volume capacity(Gi)
ledgersStorageCapacity: 10 // bookie ledgers volume capacity(Gi)
broker:
size: 3
proxy:
size: 3
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
journal-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-0 Bound pvc-8c7247e0-9a1f-4eba-9a1f-8ad10847d54d 10Gi RWO fast 2m
journal-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-1 Bound pvc-27e5b83b-6ea2-47e2-a64e-783772506959 10Gi RWO fast 84s
journal-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-2 Bound pvc-88ccca2e-0f6b-4ae9-b5c6-1b7bb57bdc69 10Gi RWO fast 62s
ledgers-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-0 Bound pvc-3afc1103-7992-41e1-9aa5-762681ac5e5d 10Gi RWO fast 2m
ledgers-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-1 Bound pvc-dd848627-6838-4cda-be77-83fe5b0b7c53 10Gi RWO fast 84s
ledgers-disk-volume-pvc-example-pulsarcluster-bookie-statefulset-2 Bound pvc-98aea962-1a7a-4c4c-a058-cef9bccacb6c 10Gi RWO fast 62s
size: 3
to size: 5
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
proxy:
size: 5
$ kubectl apply -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml
size: 3
to size: 5
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
broker:
size: 5
$ kubectl apply -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml
size: 3
to size: 5
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
name: example-pulsarcluster
spec:
bookie:
size: 5
$ kubectl apply -f deploy/crds/pulsar_v1alpha1_pulsarcluster_cr.yaml
Install kubernetes cluster
Install golang on kubernetes master node
Install operator sdkInstall Operator SDK on kubernetes master node
Clone project to the kubernetes master node(project must in GOPATH):
$ git clone https://github.com/sky-big/pulsar-operator.git
$ cd pulsar-operator
$ make start-local
$ make stop-local
$ make build
$ make image
$ make generate
(1). join
(2). pulsar-operator-dev
(3). pulsar-operator-user