Single Prometheus job for dozens of Blackbox exporters

We are stubborn on vision. We are flexible on details — Jeff Bezos

Hayk Davtyan

Published in

Geek Culture

5 min readJun 30, 2022

What is Blackbox exporter?

The Prometheus Blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC. This exporter provides metrics about endpoint status, HTTP latencies, redirect information, DNS lookups latencies as well as statistics about SSL certificates expiration. It works out-of-the-box, as it just focuses on external visibility details.

What is the story about?

This story isn’t about the installation of a Blackbox exporter, it’s more about the configuration by Prometheus side. The goal is to achieve a simple, minimal but flexible configuration by avoiding making Prometheus configuration a mess.

Imagine you have 20+ Blackbox exporters in the different locations of The World which don’t belong to any cluster or environment, just they are running as standalone applications for monitoring endpoints from different places. For example, you should monitor 100+ URLs from all locations for ensuring your website availability, latency, etc. According to documentation, a simple Prometheus job configuration looks like this:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://prometheus.io    # Probe with http.
        - https://prometheus.io   # Probe with https.
        - http://example.com:8080 # Probe with http on port 8080.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter's address.

What if you have many Blackbox exporters, URLs and multiple modules defined in the exporters’ configurations? The primitive answer is: “define jobs in the Prometheus with all targets (URLs) for each Blackbox exporter” right? Totally it will be almost 20+ jobs and more than 2000 lines of config. Definitely, you can do that, but please remember it isn’t a professional way of achieving it as we talking about Prometheus. As you see in the example of the presented job above where the module name, the targets and the address of the exporter are static, so any time when you want to change something you need to change the whole configuration. Let’s check how can be used the file_sd_config and the relabel_config features that Prometheus provides.

Configuring a single Prometheus job

At first glance, you may think that the example of the job described below is hard and complicated, but don’t worry we’ll cover that in detail.

Prometheus job: blackbox

The first important part is the file_sd_configs section which means that Prometheus will read its targets from the provided file. It’s cool because you can change the content of the file dynamically without reloading the Prometheus server. Let’s check what the file looks like.

blackbox-targets.yml

You can consider that the content of this file is similar to the part static_configs.targets which we discussed above, but the targets are a little “strange” right? Here each target contains all metadata which will be extracted in the final label set of the time series after scraping. As you can see we used the following sign :_: as a separator of the provided fields. Sure the separator can be any valid character(s). If you split the lines by the provided separator you’ll get five fields per target. The fields are the following:

<BLACKBOX_IP_PORT>:_:<MODULE>:_:<LOCATION>:_:<GEOHASH>:_:<TARGET_URL>

Blackbox exporter’s address (IP:PORT),
the module name that is defined in the Blackbox exporter config,
the city name where is located Blackbox exporter,
the value corresponding to the city’s geohash (used in the Grafana’s Geomap panel)
the target URL which needs to monitor.

The second important part is the relable_configs part. Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped. In this example are seven relabeled steps. The logic is almost the same for all steps. Just for clarification purposes, let’s cover all the steps.

Relabeling the exporter address

In this step, we should tell Prometheus which field from the target string is responsible for the__address__ label. It’s a special label that is set to the <host>:<port> address of the target.

33   # the Blackbox exporter's real hostname:port
34   - source_labels: [__address__]
35     regex: '(.*):_:.*:_:.*:_:.*:_:.*'      
36     target_label: __address__

As you noticed the value which corresponds to the address label is extracted via regex matcher. Here the (.*) is a valid RE2 regular expression whose value extracts under ${1} variable.

Input text:
1.2.3.4:9115:_:http_2xx:_:Paris:_:u09tvw0f6szye:_:http://prometheus.io
Regex:
(.*):_:.*:_:.*:_:.*:_:.*
Output:
${1} =1.2.3.4:9115

Relabeling the module

Like the previous part here is extracted the module name. The purpose of this relabeling is to have a label with the name module in the final time series.

9   # adds "module" label in the final labelset
10  - source_labels: [__address__]
11    regex: '.*:_:(.*):_:.*:_:.*:_:.*'
12    target_label: module

Input text:
1.2.3.4:9115:_:http_2xx:_:Paris:_:u09tvw0f6szye:_:http://prometheus.io
Regex:
.*:_:(.*):_:.*:_:.*:_:.*
Output:
${1} = http_2xx

Here we just pass the value of the module label to the Blackbox exporter as an HTTP query parameter that we’ve already relabeled in the previous step.

25   # passes "module" parameter to Blackbox exporter
26   - source_labels: [module]
27     target_label: __param_module

Relabeling the job label

The joblabel is something special because the real job label which is blackbox will be renamed with the location/city name where is located the appropriate exporter.

21   # rewrites "job" label with corresponding location name
22   - source_labels: [__address__]
23     regex: '.*:_:.*:_:(.*):_:.*:_:.*'
24     target_label: job

Input text:
1.2.3.4:9115:_:http_2xx:_:Paris:_:u09tvw0f6szye:_:http://prometheus.io
Regex:
.*:_:.*:_:(.*):_:.*:_:.*
Output:
${1} = Paris

Relabeling the geohash label

The geohash is the penultimate field, so it will be extracted in the following way.

13   # adds "geohash" label in the final labelset    
14   - source_labels: [__address__]      
15     regex: '.*:_:.*:_:.*:_:(.*):_:.*'      
16     target_label: geohash

Input text:
1.2.3.4:9115:_:http_2xx:_:Paris:_:u09tvw0f6szye:_:http://prometheus.io
Regex:
.*:_:.*:_:.*:_:(.*):_:.*
Output:
${1} = u09tvw0f6szye

Relabeling the instance label

In this step, the last field (the target URL) will be extracted under the instancelabel.

13   # rewrites "instance" label with corresponding URL
14   - source_labels: [__address__]
15     regex: '.*:_:.*:_:.*:_:.*:_:(.*)'
16     target_label: instance

Input text:
1.2.3.4:9115:_:http_2xx:_:Paris:_:u09tvw0f6szye:_:http://prometheus.io
Regex:
.*:_:.*:_:.*:_:.*:_:(.*)
Output:
${1} = http://prometheus.io

Like the module label here the instance label pass its value (which is http://prometheus.io) to the Blackbox exporter as an HTTP query parameter with the name target.

29   # passes "target" parameter to Blackbox exporter
30   - source_labels: [instance]
31     target_label: __param_target

As visible in the image below, the Prometheus server understands how to scrape from the targets depending on the provided configuration. (This screenshot was taken from my local computer from only the Paris endpoint).