Managing Prometheus Rules via API
APIs: Unleashing boundless potential, everywhere
Abstract
If you are someone who has ever thought about creating Prometheus rules via API, then this story is for you.
As you may know, Prometheus officially does not support creating, updating, or deleting rules via API requests. This topic has already been discussed in this GitHub issue of the Prometheus community, where the project maintainers have provided compelling arguments that prevent them from implementing this functionality in Prometheus.
In light of these challenges, I have made the decision to create an open-source project called prometheus-api that will offer the capability to manage your Prometheus rules through a REST API.
Everything you see in this story corresponds to the initial version 0.1.0 of the prometheus-api project.
Project model and prerequisites
The project enhances the native Prometheus HTTP API by providing additional features and addressing its limitations. This means that users can still access the native API endpoints provided by Prometheus, while also gaining access to the additional endpoints offered by the project. It has only a couple of requirements, which are explained below.
- The Prometheus server’s rules directory must be shared and accessible for the
prometheus-api
Since Prometheus only reads (loads) rules from files, it is necessary for the rules to be defined in the files. In this case, it will be done through the prometheus-api, which is why having a shared volume (folder) between the Prometheus server and prometheus-api is necessary.
2. The Prometheus lifecycle API must be enabled to allow requesting the /reload API
When you make an API request to create, update, or delete rules, the application calls the Prometheus server through the /reload API to trigger a reload and instruct Prometheus to reread the rules.
Deployment models
You can run the prometheus-apiserver either as a Docker container or as a sidecar in the Prometheus pod when running it in Kubernetes. Here, you can find some examples of possible deployment models that you can use to get started.
Example of how to deploy prometheus-api with Docker Compose
To deploy a prometheus-apiand Prometheus server, you need to have a docker-compose.yml file similar to the one provided below.
Before starting the Docker Compose, create a project directory and download a prometheus.yml file that includes the configuration section specifying from where Prometheus should load the rule files.
$ mkdir prometheus-api-project
$ cd prometheus-api-project
$ curl -LO https://raw.githubusercontent.com/hayk96/prometheus-api/main/docs/examples/docker/prometheus.ymlCreate docker-compose.yml file
$ cat <<EOF > docker-compose.yml
version: "3.5"
services:
prometheus:
image: prom/prometheus:v2.44.0
container_name: prometheus
ports:
- "9090:9090"
volumes:
- "./rules:/etc/prometheus/rules"
- "./prometheus.yml:/etc/prometheus/prometheus.yml"
command:
- --config.file=/etc/prometheus/prometheus.yml
- --web.enable-lifecycle
prometheus-api:
image: hayk96/prometheus-api:v0.1.0
container_name: prometheus-api
ports:
- "5000:5000"
volumes:
- "./rules:/app/rules:rw"
command:
- --prom.addr=http://prometheus:9090
- --rule.path=/app/rules
EOFOnce you have brought up the Docker Compose, you will see both running services.
$ docker compose up -d
$ docker compose ps
NAME COMMAND SERVICE STATUS PORTS
prometheus "/bin/prometheus --c…" prometheus running 0.0.0.0:9090->9090/tcp
prometheus-api "python main.py --pr…" prometheus-api running 0.0.0.0:5000->5000/tcpAs it’s visible the prometheus-api container is listening on port 5000. Let’s open http://localhost:5000 in the browser.
Wow, as you can see, it’s just the Prometheus UI, right? The reason is that the prometheus-apiserver also acts as a proxy server, redirecting all requests to the downstream Prometheus server. When you try to open the URL http://localhost:5000/docs, you will see the default documentation page of the FastAPI — Swagger UI.
Now it’s time to make some requests to create our first Prometheus alerting rule via the API. We need to call the /api/v1/rules endpoint using the POST method.
$ curl -i -XPOST 'http://localhost:5000/api/v1/rules' \
--header 'Content-Type: application/json' \
--data '{
"data": {
"groups": [
{
"name": "ServiceHealthAlerts",
"rules": [
{
"alert": "HighCPUUsage",
"expr": "sum(rate(cpu_usage{job=\"webserver\"}[5m])) > 0.8",
"for": "5m",
"labels": {
"severity": "warning"
},
"annotations": {
"summary": "High CPU Usage Detected",
"description": "The CPU usage for the web server is {{ $value }}% for the last 5 minutes."
}
}
]
}
]
}
}'After successfully completing the request, the server will return a response similar to the one shown below, where the zwhsjnggqaouehw.yml is the randomly generated filename created by the Prometheus API server.
*Note that the filename in your example is different from the example below.
HTTP/1.1 201 Created
content-length: 99
content-type: application/json
{"status":"success","message":"The rule was created successfully","file":"zwhsjnggqaouehw.yml"}Open the Prometheus UI and verify whether the requested rule has been successfully created.
As you can see, the rule has been created successfully. You can also delete the rule using the appropriate API. To delete the created rule, you need to call the /api/v1/rules/{file} endpoint using the DELETE method. Here the {file} parameter represents the resource name (filename) in this case it’s zwhsjnggqaouehw.yml.
$ curl -i -XDELETE 'http://localhost:5000/api/v1/rules/zwhsjnggqaouehw.yml'The successful response should be exactly like this.
HTTP/1.1 204 No Content
content-type: application/jsonConclusion
If you are interested in working with the prometheus-api, you can refer to the complete API documentation here, where you can find comprehensive information and instructions on how to use it.
Thanks for reading. I hope this story was helpful. If you are interested, check out my other Medium articles.
