Skip to content

Installing Kafka

This section explains the steps to install Kafka using Helm in a Kubernetes environment. It covers adding the Bitnami Helm repository, configuring Kafka and ZooKeeper parameters, enabling monitoring and deploying Kafka with custom parameters for your infrastructure.

To install Kafka, follow the steps:

  1. Add the Bitnami Helm repository by running the following command.

    helm repo add bitnami https://charts.bitnami.com/bitnami
    
  2. Download the specific version (25.1.7) of the Kafka Helm chart from the Bitnami repository.

    helm fetch bitnami/kafka --version 25.1.7
    
  3. Update the following parameters according to the respective software platform.

    • Global parameter
    Name Description Value
    global.storageClass Global StorageClass for Persistent Volume(s) "longhorn-ssd"
    • Kafka parameter
    Name Description Value
    heapOpts Kafka Java Heap size -Xmx2048m -Xms2048m
    • Controller-eligible statefulset parameter:
    Name Description Value
    controller.replicaCount Number of Kafka controller-eligible nodes 0
    • Broker-only statefulset parameters:
    Name Description Value
    broker.replicaCount Number of Kafka broker-only nodes 3
    broker.heapOpts Kafka Java Heap size for broker-only nodes -Xmx2048m -Xms2048m
    broker.resources.limits The resources limits for the container {"cpu": "1", "memory": "8.5Gi"}
    broker.resources.requests The requested resources for the container {"cpu": "250m", "memory": "2.5Gi"}
    broker.affinity Affinity for pod assignment {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "iaas", "operator": "In", "values": ["true"]}]}]}}}
    broker.tolerations Tolerations for pod assignment [{"effect": "NoSchedule", "key": "domain", "operator": "Equal", "value": "iaas"}]
    broker.persistence.enabled Enable Kafka data persistence using PVC, note that ZooKeeper persistence is unaffected true
    broker.persistence.size PVC Storage Request for Kafka data volume 1Ti
    • Metrics parameters:
    Name Description Value
    metrics.kafka.enabled Whether or not to create a standalone Kafka exporter to expose Kafka metrics true
    metrics.kafka.resources.limits The resources limits for the container {"cpu": "200m", "memory": "256Mi"}
    metrics.kafka.resources.requests The requested resources for the container {"cpu": "100m", "memory": "128Mi"}
    metrics.kafka.affinity Affinity for pod assignment {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "iaas", "operator": "In", "values": ["true"]}]}]}}}
    metrics.kafka.tolerations Tolerations for pod assignment [{"effect": "NoSchedule", "key": "domain", "operator": "Equal", "value": "iaas"}]
    metrics.jmx.enabled Whether or not to expose JMX metrics to Prometheus true
    metrics.jmx.resources.limits The resources limits for the JMX exporter container {"cpu": "200m", "memory": "256Mi"}
    metrics.jmx.resources.requests The requested resources for the JMX exporter container {"cpu": "100m", "memory": "128Mi"}
    metrics.serviceMonitor.enabled If true, creates a Prometheus Operator ServiceMonitor (requires metrics.kafka.enabled or metrics.jmx.enabled to be true) true
    • KRaft chart parameter:
    Name Description Value
    kraft.enabled Switch to enable or disable the KRaft mode for Kafka false
    • ZooKeeper chart parameters:
    Name Description Value
    zookeeper.enabled Switch to enable or disable the ZooKeeper helm chart. Must be false if you use KRaft mode. true
    zookeeper.replicaCount Number of ZooKeeper nodes 3
    zookeeper.persistence.enabled Enable persistence on ZooKeeper using PVC(s) true
    zookeeper.persistence.size Persistent Volume size 5Gi
  4. Navigate to the following file to define the broker template.

    kafka/templates/broker/statefulset.yaml
    

    Set the following broker template parameters:

    metadata.name

    metadata:
      name:
        - \{\{ printf "%s-broker" (include "common.names.fullname" .) \}\}
        - \{\{ printf "%s" (include "common.names.fullname" .) \}\}
    
  5. Install Kafka in the mdsp-bk-kafka namespace by running the following command.

    helm -n mdsp-bk-kafka install kafka
    

Last update: January 31, 2025