I couldn’t find the right answer to the problem with the prometheus.io/scrape annotation not working in Kubernetes, so I decided to dig into the original Prometheus Helm chart. For those facing the same issue, here’s an explanation.

Understanding Prometheus Configurations in Kubernetes

First, it’s important to note that Prometheus can be configured in multiple ways within Kubernetes. One common method is using the Custom Resource Definition (CRD) called ServiceMonitor. In this setup, the Prometheus Operator continuously monitors resources specified by ServiceMonitor objects.

•   serviceMonitorSelector Parameter: This parameter in the Prometheus Operator configuration selects which ServiceMonitor resources to consider.

serviceMonitorSelector: matchLabels: team: frontend

•   No Default Annotations: By default, services or pods aren’t monitored based on annotations alone. You need to:
•   Create a ServiceMonitor: Define a ServiceMonitor resource that matches your services or pods.
•   Set Appropriate Labels: Ensure your services or pods have labels that match the matchLabels in your ServiceMonitor.

However, the default kube-prometheus-stack Helm chart doesn’t create a ServiceMonitor for your deployments out of the box.

The Origin of prometheus.io/scrape Annotation

This brings up the question:

Where does the prometheus.io/scrape annotation come from, and how can I use it?

The answer lies in the original Prometheus Helm chart, which you can find here. Unlike the kube-prometheus-stack, this Helm chart doesn’t rely on Prometheus CRDs. Instead, it:

•   Deploys Prometheus Directly: Runs Prometheus in a pod with manual configurations.
•   Uses kubernetes_sd_configs: Specifies how Prometheus should discover services.

kubernetes_sd_configs:

• role: endpoints

This tells Prometheus to use the endpoints role, allowing it to scrape targets based on annotations like prometheus.io/scrape.

•   Relabeling Configurations: Includes additional settings to manipulate labels and target metadata.

How to Resolve the Issue

If you’re using kube-prometheus-stack, you have two main options:

1.  Set Up a ServiceMonitor
•   Create a ServiceMonitor Resource: Define it to match your services or pods.
•   Adjust serviceMonitorSelector: Ensure the Prometheus Operator picks up your ServiceMonitor.

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: my-service-monitor labels: team: frontend spec: selector: matchLabels: app: my-app endpoints: - port: web

2.  Modify Prometheus Configuration
•   Include endpoints Role: Adjust your Prometheus config to use the endpoints role like in the original Helm chart.
•   Leverage Annotations: This allows you to use annotations like prometheus.io/scrape without needing ServiceMonitor.

prometheus: prometheusSpec: additionalScrapeConfigs: - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true

Summary

If the prometheus.io/scrape annotation isn’t working with kube-prometheus-stack:

•   Use a ServiceMonitor: It’s the preferred method when using the Prometheus Operator.
•   Copy Configuration from Original Helm Chart: Adjust your Prometheus configuration to manually include endpoint discovery based on annotations.

By following these steps, you should be able to enable Prometheus to scrape your services or pods as expected.