Installing Kafka¶
This section explains the steps to install Kafka using Helm in a Kubernetes environment. It covers adding the Bitnami Helm repository, configuring Kafka and ZooKeeper parameters, enabling monitoring and deploying Kafka with custom parameters for your infrastructure.
To install Kafka, follow the steps:
-
Add the Bitnami Helm repository by running the following command.
helm repo add bitnami https://charts.bitnami.com/bitnami
-
Download the specific version (25.1.7) of the Kafka Helm chart from the Bitnami repository.
helm fetch bitnami/kafka --version 25.1.7
-
Update the following parameters according to the respective software platform.
- Global parameter
Name Description Value global.storageClass
Global StorageClass for Persistent Volume(s) " longhorn-ssd
"- Kafka parameter
Name Description Value heapOpts
Kafka Java Heap size -Xmx2048m -Xms2048m - Controller-eligible statefulset parameter:
Name Description Value controller.replicaCount
Number of Kafka controller-eligible nodes 0
- Broker-only statefulset parameters:
Name Description Value broker.replicaCount
Number of Kafka broker-only nodes 3
broker.heapOpts
Kafka Java Heap size for broker-only nodes -Xmx2048m -Xms2048m
broker.resources.limits
The resources limits for the container {"cpu": "1", "memory": "8.5Gi"} broker.resources.requests
The requested resources for the container {"cpu": "250m", "memory": "2.5Gi"} broker.affinity
Affinity for pod assignment {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "iaas", "operator": "In", "values": ["true"]}]}]}}} broker.tolerations
Tolerations for pod assignment [{"effect": "NoSchedule", "key": "domain", "operator": "Equal", "value": "iaas"}] broker.persistence.enabled
Enable Kafka data persistence using PVC, note that ZooKeeper persistence is unaffected true
broker.persistence.size
PVC Storage Request for Kafka data volume 1Ti
- Metrics parameters:
Name Description Value metrics.kafka.enabled
Whether or not to create a standalone Kafka exporter to expose Kafka metrics true
metrics.kafka.resources.limits
The resources limits for the container {"cpu": "200m", "memory": "256Mi"} metrics.kafka.resources.requests
The requested resources for the container {"cpu": "100m", "memory": "128Mi"} metrics.kafka.affinity
Affinity for pod assignment {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "iaas", "operator": "In", "values": ["true"]}]}]}}} metrics.kafka.tolerations
Tolerations for pod assignment [{"effect": "NoSchedule", "key": "domain", "operator": "Equal", "value": "iaas"}] metrics.jmx.enabled
Whether or not to expose JMX metrics to Prometheus true
metrics.jmx.resources.limits
The resources limits for the JMX exporter container {"cpu": "200m", "memory": "256Mi"} metrics.jmx.resources.requests
The requested resources for the JMX exporter container {"cpu": "100m", "memory": "128Mi"} metrics.serviceMonitor.enabled
If true
, creates a Prometheus Operator ServiceMonitor (requiresmetrics.kafka.enabled or metrics.jmx.enabled
to betrue
)true
- KRaft chart parameter:
Name Description Value kraft.enabled
Switch to enable or disable the KRaft mode for Kafka false
- ZooKeeper chart parameters:
Name Description Value zookeeper.enabled
Switch to enable or disable the ZooKeeper helm chart. Must be false if you use KRaft mode. true
zookeeper.replicaCount
Number of ZooKeeper nodes 3
zookeeper.persistence.enabled
Enable persistence on ZooKeeper using PVC(s) true
zookeeper.persistence.size
Persistent Volume size 5Gi
-
Navigate to the following file to define the broker template.
kafka/templates/broker/statefulset.yaml
Set the following broker template parameters:
metadata: name: - \{\{ printf "%s-broker" (include "common.names.fullname" .) \}\} - \{\{ printf "%s" (include "common.names.fullname" .) \}\}
-
Install Kafka in the
mdsp-bk-kafka
namespace by running the following command.helm -n mdsp-bk-kafka install kafka
Last update: January 31, 2025