Kubernetes Cluster Build Series

This article is part of a step-by-step series on building a production-style Kubernetes cluster from scratch.

Series so far:

• Part 0 – Before You Begin (Prerequisites & Planning)

• Part 1 – Preparing Linux Nodes for Kubernetes

• Part 2 – Installing and Configuring containerd Runtime

• Part 3 – Installing Kubernetes and Bootstrapping the Cluster (kubeadm + Calico)

• Part 4 – Understanding Kubernetes DNS and CoreDNS Stability

• Part 5 – Deploying Persistent Storage with Longhorn

• Part 6 – Object Storage in Kubernetes After MinIO

• Part 7 - Deploying Identity and Access Management in Kubernetes with Keycloak

• Part 8 – Deploying Kafka in Kubernetes with Strimzi (Current Article)

Each part builds on the previous one, so it is recommended to follow the series in order.

---

Why Kafka matters in distributed systems and Kubernetes

At this point in the series, the cluster already has:

• Networking

• Storage

• Object storage strategy

• Identity management

The next major capability is messaging.

Modern platforms often need applications to exchange data asynchronously. Instead of one application calling another directly, messages are written into a broker and consumed independently.

This is where Apache Kafka becomes useful.

Kafka is commonly used for:

• Event streaming

• Log pipelines

• Real-time analytics

• Application integration

• Decoupled microservices

---

Why use Strimzi

Running Kafka directly in Kubernetes can be difficult because Kafka requires:

• Multiple brokers

• Coordination between components

• Persistent storage

• Topic management

Strimzi simplifies this by providing a Kubernetes operator for Kafka.

Strimzi manages:

• Kafka clusters

• Topics

• Connectors

• Bridge services

• Rolling upgrades

This makes Kafka feel much more “Kubernetes-native”.

---

Downloading Strimzi

Before deploying Strimzi, download the release archive and extract it:

wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.51.0/strimzi-0.51.0.zip

unzip strimzi-0.51.0.zip

cd strimzi-0.51.0

This creates a local directory containing:

• Installation manifests

• Kafka examples

• Kafka Connect examples

• Kafka Bridge examples

---

Creating a namespace for Kafka

Kafka components should run in a dedicated namespace.

Example:

kubectl create namespace kafka-system

You may choose any namespace name, but keeping Kafka isolated is recommended.

---

Installing the Strimzi Operator

Apply the operator manifests from the extracted archive.

Typical flow:

kubectl apply -f install/cluster-operator -n kafka-system

In some environments, additional RBAC manifests may need to be applied before the operator.

For example:

kubectl apply -f install/strimzi-admin/

Then verify:

kubectl get pods -n kafka-system

The Strimzi operator pod should become Running.

---

Creating a Kafka Cluster

Once the operator is running, deploy a Kafka cluster using one of the example manifests.

Example:

kubectl apply -f examples/kafka/kafka-persistent.yaml -n kafka-system

This typically creates:

• Kafka brokers

• Kafka controllers

• Persistent volumes

• Services

Kafka may take several minutes to start because brokers require storage and cluster coordination.

Verify:

kubectl get pods -n kafka-system

Wait until all Kafka-related pods are in Running state.

---

Creating a Kafka Topic

Once Kafka is running, create a topic.

Example:

kubectl apply -f mytopic.yaml -n kafka-system

A typical Kafka topic manifest contains:

• Topic name

• Number of partitions

• Replication factor

Example conceptually:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: application-events
spec:
  partitions: 3
  replicas: 3

This creates a durable topic that applications can publish to and consume from.

---

Deploying Kafka Connect

Kafka Connect is used to move data between Kafka and external systems.

Typical examples:

• Database to Kafka

• Kafka to object storage

• Kafka to Elasticsearch

Deploy Kafka Connect using the provided example manifest:

kubectl apply -f examples/connect/kafka-connect.yaml -n kafka-system

Verify:

kubectl get pods -n kafka-system

Kafka Connect should appear as another running pod.

---

Deploying Kafka Bridge

Kafka Bridge allows HTTP clients to interact with Kafka.

This is useful when applications cannot use Kafka client libraries directly.

Deploy Kafka Bridge:

kubectl apply -f examples/bridge/kafka-bridge.yaml -n kafka-system

Kafka Bridge exposes Kafka through REST APIs.

This can be useful for:

• Web applications

• Simple integrations

• External systems

---

Verifying the Kafka Platform

After deployment, verify:

kubectl get pods -n kafka-system
kubectl get svc -n kafka-system

You should see:

• Operator pod

• Kafka brokers

• Kafka Connect

• Kafka Bridge

Everything should be in Running state.

---

Common issues to watch for

Typical Kafka-on-Kubernetes problems include:

• Pods stuck in Pending due to missing storage

• Brokers failing because of insufficient memory

• Kafka topics not creating because operator is not ready

• Kafka Connect failing because connectors are missing

Kafka is one of the heavier components in this series, so ensure your cluster has enough CPU, memory, and storage before deploying it.

---

How Kafka fits into the platform

Kafka now completes the core platform architecture built throughout this series.

The cluster now supports:

• Stateful storage

• Object storage

• Identity

• Messaging

Applications can now:

• Authenticate with Keycloak

• Store files in object storage

• Publish and consume events through Kafka

This is the foundation of a modern internal platform.

---

What’s next

In the next and final article, we will bring everything together and look at the complete architecture of the platform, how the components interact, and what improvements could be made in a production environment.