Monitoring Prometheus Operator¶
This section explains the steps to configure and monitor the Prometheus Operator on a Kubernetes cluster. It includes customizing the Helm chart with tolerations, affinity, storage settings and configuring Prometheus to monitor external services.
To configure Prometheus Operator, follow the steps:
-
Clone the Prometheus Operator repository.
https://github.com/prometheus-operator/prometheus-operator
-
Update the Helm Chart with the following parameters in the
values.yaml
file.Note: No additional configurations are required for the node exporter, as it is deployed as a
DaemonSet
.-
Update tolerations and affinity for Grafana,
Kube-state-metrics
, Prometheus and the Prometheus Operator.tolerations: - key: "domain" operator: "Equal" value: "devops" effect: "NoSchedule" affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "devops/prometheus" operator: "In" values: - "true"
-
Add storage specifications for Prometheus.
storageSpec: volumeClaimTemplate: spec: storageClassName: longhorn accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi
-
Add job configurations to monitor external services outside of Kubernetes.
- job_name: 'hbase-master' scrape_interval: 60s honor_labels: true static_configs: - targets: ['<ip-address>','<ip-address>.13:17001','<ip-address>.193:17002','<ip-address>:17002'] - job_name: node-exporter scrape_interval: 30s honor_labels: true static_configs: - targets: ['<ip-address>:9100','<ip-address>:9100','<ip-address>:9100']
-
-
Enable or disable services as per the configuration in
values.yaml
.- Install the Prometheus Operator using Helm:
helm install <name> ./prometheus_operator -n <namespace>
- Upgrade the existing Prometheus Operator deployment:
helm upgrade <name> ./prometheus_operator -n <namespace>
-
Configure and expose Grafana and Prometheus as Ingress resources for external access.
- Install the Prometheus Operator using Helm:
-
Enable metrics for the backend service and add a
ServiceMonitor
.Add the label release:
<name>
in theServiceMonitor
to associate it with the release name from step 3. -
Verify the status of the monitor service within the Prometheus dashboard to ensure that it is scraping the required metrics.
-
Use the Prometheus UI or CLI to check the service status on Prometheus.
-
Add the corresponding dashboard.