Transforming remote JSON into Prometheus metrics

Image by Hayk Davtyan (author)

Introduction.

Have you ever thought to turn JSON objects into Prometheus format metrics? This point can be implemented via prometheus-json-exporter. that scrapes remote JSON by JSONPath.

About this tutorial.

In this tutorial, we’ll learn how to deploy and configure json_exporter for a public JSON endpoint and use it with Prometheus. The deployment method that we’ll use is the Docker compose, although it can be done in the Kubernetes environment as well.

As a JSON endpoint, we’ll use a public API provided by CoinStats. As you already guessed it’ll be about cryptocurrencies. Really cool, yes? It means that we are going to turn real-time information of cryptos into Prometheus-based metrics which will allow storing data historically. Let’s build it!

Creating a working directory and config files for this tutorial.

I’m going to use the /tmp directory for this project, please use another directory for avoiding losing the files in case of restarting the system.

$ mkdir -p /tmp/json_exporter_tutorial/examples

Before creating the rest configuration files for the services, let’s see how looks like a single JSON object of this: api.coinstats.app/public/v1/coins API.

{
  "coins": [
    {
      "id": "bitcoin",
      "icon": "https://static.coinstats.app/coins/Bitcoin6l39t.png",
      "name": "Bitcoin",
      "symbol": "BTC",
      "rank": 1,
      "price": 40224.98822527244,
      "priceBtc": 1,
      "volume": 396892467427897.7,
      "marketCap": 762621770614.047,
      "availableSupply": 18958906,
      "totalSupply": 21000000,
      "priceChange1h": 0.58,
      "priceChange1d": 0,
      "priceChange1w": -4.52,
      "websiteUrl": "http://www.bitcoin.org",
      "twitterUrl": "https://twitter.com/bitcoin",
      "exp": [
        "https://blockchair.com/bitcoin/",
        "https://btc.com/",
        "https://btc.tokenview.com/"
      ]
    }
  ]
}

As we already know the structure of the JSON object we can tell to exporter what and how should it parse the entire object for converting them into Prometheus time-series.

Defining metric names and labels.

Depending on the JSON object the mandatory labels that will exist in each time-series are: id , name and symbol. The metrics will represent the corresponding values of the following keys: rank, price, priceBtc, volume, marketCap, availableSupply, totalSupply, priceChange[1h,1d,1w].

So the metric names we’re going to create will be:

• cryptocurrency_rank
• cryptocurrency_price
• cryptocurrency_price_as_btc
• cryptocurrency_volume
• cryptocurrency_market_cap
• cryptocurrency_available_supply
• cryptocurrency_total_supply
• cryptocurrency_price_change1h
• cryptocurrency_price_change1d
• cryptocurrency_price_change1w

Create the configuration file for the json_exporter with the following content.

$ touch /tmp/json_exporter_tutorial/examples/config.yml

config.yml - configuration file for json_exporter

The following configuration represents the steps parsing of a JSON object with JSONPath.

Create a configuration file for Prometheus with the following content.

$ touch /tmp/json_exporter_tutorial/examples/prometheus.yml

prometheus.yml - Prometheus job for json_exporter

As you noticed we are going to use three targets that differ in ?currency query parameters. With the help of mentioned parameter, the response will be appropriate to that currency, so in this case, we’ll have time-series for three different types of currencies USD, EUR, and AMD. The following section of relabel_configs will create thecurrency label corresponding to the value of the ?currency HTTP query parameter.

- source_labels: [__address__]
  regex: ^http.+currency=([A-Z]{3})
  action: replace
  target_label: currency
  replacement: ${1}

And the last file that we need to create is the docker-compose.yml. Create it with the following content described below.

$ touch /tmp/json_exporter_tutorial/docker-compose.yml

docker-compose.yml

Finally, it’s time to start the services with Docker compose.

$ cd /tmp/json_exporter_tutorial
$ docker-compose up -d

As the services have been successfully started, let’s open the Prometheus web interface http://localhost:9090/targets and check the state of the targets. The three targets are UP, which means everything has been done correctly.

Img. 1 Prometheus targets

Doing some queries.

Since the Prometheus scrapes successfully from the targets, let’s see what the time-series look like.

• bottomk(10, avg by (name) (cryptocurrency_rank)) - returns top 10 cryptocurrencies by ranking.

Img. 2 Top 10 cryptocurrencies

• sum by (currency,name) (cryptocurrency_price{symbol="ETH"}) - returns the current price of ETH per currency.

Img. 3 ETH current price per currency

Conclusion.

Definitely, we can say that json_exporter is a very useful tool, it’s easy to configure and integrate with the Prometheus stack. Thanks to the creators and maintainers of this exporter.

Thanks for reading. I hope this story was helpful. If you are interested, check out my other Medium articles.