Skip to content

Monitoring Prometheus Operator

This section explains the steps to configure and monitor the Prometheus Operator on a Kubernetes cluster. It includes customizing the Helm chart with tolerations, affinity, storage settings and configuring Prometheus to monitor external services.

To configure Prometheus Operator, follow the steps:

  1. Clone the Prometheus Operator repository.

    https://github.com/prometheus-operator/prometheus-operator
    
  2. Update the Helm Chart with the following parameters in the values.yaml file.

    Note: No additional configurations are required for the node exporter, as it is deployed as a DaemonSet.

    • Update tolerations and affinity for Grafana, Kube-state-metrics, Prometheus and the Prometheus Operator.

      tolerations:
      - key: "domain"
      operator: "Equal"
      value: "devops"
      effect: "NoSchedule"
      
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "devops/prometheus"
      operator: "In"
      values:
                - "true"
      
    • Add storage specifications for Prometheus.

      storageSpec:
        volumeClaimTemplate:
          spec:
            storageClassName: longhorn
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 50Gi
      
    • Add job configurations to monitor external services outside of Kubernetes.

      - job_name: 'hbase-master'
      scrape_interval: 60s
        honor_labels: true
        static_configs:
        - targets: ['<ip-address>','<ip-address>.13:17001','<ip-address>.193:17002','<ip-address>:17002']
      - job_name: node-exporter
        scrape_interval: 30s
        honor_labels: true
        static_configs:
        - targets: ['<ip-address>:9100','<ip-address>:9100','<ip-address>:9100']
      
  3. Enable or disable services as per the configuration in values.yaml.

    1. Install the Prometheus Operator using Helm: helm install <name> ./prometheus_operator -n <namespace>
    2. Upgrade the existing Prometheus Operator deployment: helm upgrade <name> ./prometheus_operator -n <namespace>
    3. Configure and expose Grafana and Prometheus as Ingress resources for external access.

      Hostname

      Promo

  4. Enable metrics for the backend service and add a ServiceMonitor.

    Add the label release: <name> in the ServiceMonitor to associate it with the release name from step 3.

    Service

  5. Verify the status of the monitor service within the Prometheus dashboard to ensure that it is scraping the required metrics.

    Metrics

  6. Use the Prometheus UI or CLI to check the service status on Prometheus.

    Target

  7. Add the corresponding dashboard.

    Monitor


Last update: January 27, 2025