Kubernetes Service Discovery in Prometheus

Prometheus seems to be the most popular monitoring system for kubernetes these days. It is impossibile for us to use static scrape targets in prometheus config for kubernetes metrics, as things varies all the time in kubernetes. That’s where service discovery comes in.

Prometheus provides service discovery for kubernetes node, service, pod, endpoint and ingress. Take pod as example, we can pull from all the pods in kubernetes by put this in config:

1
2
3


- job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod

The config means pull metrics from https://${POD_IP}:${POD_PORT}/metrics of all the pods in kubernetes, where POD_IP and POD_PORT can be found in pod spec. This works in theory but practically, we don’t want to scrape from all the pods and sometimes it provides metrics in specific port and path. How do we implement a switch for scraping and specify scrape address for each pod? The answer is relabelling, let’s take a look at the solution gives by kubernetes (https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/prometheus/prometheus-configmap.yaml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


- job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (.+)

It is hard to understand what’s going on here if you are not familiar with relabel config, let’s break it down.

First of all, we can use it by special pod annotations:

1
2
3
4
5


annotations:
  "prometheus.io/scrape":"true",
  "prometheus.io/path":"/mypath/metrics",
  "prometheus.io/port":"8080",
  "prometheus.io/scheme":"http"

Labels begin with __ are special labels used inside prometheus. The scrape address will be represents by ${__scheme__}://${__address__}/${__metrics_path__} which is http://${POD_IP}:8080/mypath/metrics. Labels begin with __meta_kubernetes_pod_annotation represent pod annotations, and relabel_configs works one by one:

Continue if prometheus.io/scrape is set to true, else ignore the pod;
Replace value of __metrics_path__by value of prometheus.io/path;
Change port in __address__ by value of prometheus.io/port;
Replace value of __scheme__by value of prometheus.io/scheme;

For other resources like nodes, services or endpoints, it works the same way as pods. The details of configuration can be found at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config.

Contents