kafka setup in k8s cluster using strimizi

To setup a kafka cluster in k8s is pretty easy with Strimizi. 

We can start by creating a namespace call kafka using the following command

kubectl create namespace kafka

Then we setup stirmizi CRDs

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

And finally installing the cluster

kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-single-node.yaml -n kafka 

If everything goes well, then you should have the followings 

You can run the following command to check for your kakfa instance

k get KafkaNodePool -A

k get Kafka -A

Send and receive messages

To test out the consumer, run the following command and you can keep on entering messages by pressing enter. It will continuously accept inputs until you press Ctrl+break

kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.46.0-kafka-4.0.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic

To test out the consumer, run the following command 

kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.46.0-kafka-4.0.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning

Next setting up the UI

It would be nice to be able to view your kakfa cluster via UI. In this setup, we are going to use Provectus to do this. To do that, we need the following yaml

apiVersion: v1

kind: Service

metadata:

  name: kafka-ui

  namespace: kafka

spec:

  type: ClusterIP

  ports:

    - port: 8080

      targetPort: 8080

  selector:

    app: kafka-ui

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: kafka-ui

  namespace: kafka

spec:

  replicas: 1

  selector:

    matchLabels:

      app: kafka-ui

  template:

    metadata:

      labels:

        app: kafka-ui

    spec:

      containers:

        - name: kafka-ui

          image: provectuslabs/kafka-ui:latest

          ports:

            - containerPort: 8080

          env:

            - name: KAFKA_CLUSTERS_0_NAME

              value: "my-cluster"

            - name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS

              value: "my-cluster-kafka-bootstrap:9092" # Adjust to your Kafka service

Once this has been completed, we can do a port-forward

kubectl port-forward svc/kafka-ui -n kafka 8080:8080

And this is what the UI looks like. It shows basic information about the cluster and topics. 

That's it!