Pushgateway and Prometheus Transform Data Collection and Analysis

Introduction

Pushgateway provides a straightforward solution for scenarios where metrics need to be collected from various sources that do not have a long-running lifespan or do not expose a direct Prometheus endpoint. By acting as an intermediary, Pushgateway enables the collection of time series data from batch jobs, scripts, and short-lived processes. This capability is particularly valuable when dealing with ephemeral tasks executed sporadically or for a limited duration.

One of the key benefits of Pushgateway is its ability to store time series data for a relatively short period. This design choice aligns well with the nature of batch jobs, which often have defined start and end times. Once a batch job completes, the associated metrics can be pushed to the Pushgateway, allowing Prometheus to scrape and store them for analysis. Subsequently, the Pushgateway can be reset to accept new data for the next execution of the job.

Moreover, Pushgateway extends its usefulness beyond batch jobs. It can also effectively handle scenarios involving services with a high turnover rate. In dynamic environments where service instances frequently come and go, collecting and consolidating metrics from these transient components can be challenging. Pushgateway acts as a central hub, facilitating the storage and retrieval of metrics from these services, ensuring the data remains available for analysis and monitoring.

By leveraging Pushgateway, organizations can aggregate metrics from disparate sources and present them within Prometheus for comprehensive analysis. This centralized approach allows for a holistic view of the system’s performance, making it easier to identify trends, troubleshoot issues, and make data-driven decisions. Pushgateway is a flexible and versatile tool in the Prometheus ecosystem, empowering users to efficiently collect and manage time series data from diverse and evolving sources.

Practical Example

Here’s a practical example of how to use Pushgateway with Prometheus:

Assuming that you have Prometheus already set up and running. Let’s install Pushgateway

wget https://github.com/prometheus/pushgateway/releases/download/v1.5.1/pushgateway-1.5.1.linux-amd64.tar.gz
tar xvzf pushgateway-1.5.1.linux-amd64.tar.gz
cd pushgateway-1.5.1.linux-amd64/ 
./pushgateway &

wget https://github.com/prometheus/pushgateway/releases/download/v1.5.1/pushgateway-1.5.1.linux-amd64.tar.gz

tar xvzf pushgateway-1.5.1.linux-amd64.tar.gz

cd pushgateway-1.5.1.linux-amd64/

./pushgateway &

2. Let’s create a script that generates metrics and pushes them to Pushgateway. Before that, we first install prometheus_client,

sudo apt install python3-pip
pip3  install prometheus_client

1 2	sudo apt install python3-pip pip3 install prometheus_client

First, let’s create a simple Python script that generates some random metrics using the Prometheus Python client library. This script generates random values for two metrics, “test_metric_1” and “test_metric_2”, with a random sleep time between each push to simulate different times of metrics.

import random
import time
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
# Create a registry for the metrics
registry = CollectorRegistry()
# Create two gauges for the metrics
gauge_1 = Gauge('test_metric_1', 'Test Metric 1', registry=registry)
gauge_2 = Gauge('test_metric_2', 'Test Metric 2', registry=registry)
# Generate random values and push them to Pushgateway every few seconds
while True:
    value_1 = random.randint(1, 100)
    value_2 = random.randint(1, 100)
    gauge_1.set(value_1)
    gauge_2.set(value_2)
    push_to_gateway('localhost:9091', job='test_job', registry=registry)
    time.sleep(random.randint(1, 10))

import random

import time

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

# Create a registry for the metrics

registry = CollectorRegistry()

# Create two gauges for the metrics

gauge_1 = Gauge('test_metric_1', 'Test Metric 1', registry=registry)

gauge_2 = Gauge('test_metric_2', 'Test Metric 2', registry=registry)

# Generate random values and push them to Pushgateway every few seconds

while True:

value_1 = random.randint(1, 100)

value_2 = random.randint(1, 100)

gauge_1.set(value_1)

gauge_2.set(value_2)

push_to_gateway('localhost:9091', job='test_job', registry=registry)

time.sleep(random.randint(1, 10))

3. Save this script as “pushgateway_test.py” and run it in a terminal window with the command:

python pushgateway_test.py

1	python pushgateway_test.py

4. Now let’s configure Prometheus to scrape the metrics from Pushgateway. Add the following configuration to your Prometheus configuration file:

scrape_configs:
  - job_name: 'test_job'
    honor_labels: true
    metrics_path: '/metrics'
    scheme: 'http'
    static_configs:
      - targets: ['localhost:9091']

scrape_configs:

- job_name: 'test_job'

honor_labels: true

metrics_path: '/metrics'

scheme: 'http'

static_configs:

- targets: ['localhost:9091']

This configuration tells Prometheus to scrape metrics from the Pushgateway instance running on localhost:9091.

5. After saving the configuration file, restart Prometheus for the changes to take effect.

6. Now you can access the Pushgateway metrics in Prometheus by querying the “pushgateway” job. For example, you can use the PromQL query language to calculate the average value of “test_metric_1” over the last 5 minutes:

avg_over_time(test_metric_1[5m])

1	avg_over_time(test_metric_1[5m])

This will return the average value of “test_metric_1” over the last 5 minutes.

push

push2

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Conclusion

Pushgateway can be a powerful tool for collecting and aggregating metrics from different processes and services and presenting them in Prometheus for monitoring and analysis. By using Pushgateway in conjunction with Prometheus, you can gain valuable insights into the performance and behavior of your systems.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft Gold Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding Pushgateway, Prometheus, I will get back to you quickly.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. What is the best use case for Pushgateway?

ANS: – Pushgateway is a useful tool for collecting and aggregating metrics from batch jobs, scripts, or other short-lived processes that don’t expose a Prometheus endpoint. It’s designed to store time series data for relatively short periods, such as the duration of a batch job, and then be reset to accept new data for the next run of the job. Pushgateway can also store data from services with a high turnover rate, where service instances may come and go frequently.

2. How does Pushgateway differ from Prometheus exporters?

ANS: – Prometheus exporters are used to exposing metrics from services or applications that run continuously and can be scraped by Prometheus at regular intervals. In contrast, Pushgateway is designed to store metrics from batch jobs, scripts, or other short-lived processes that don’t run continuously and don’t expose a Prometheus endpoint. Instead of having Prometheus scrape the metrics from the jobs or scripts, the metrics are pushed to the Pushgateway for temporary storage until Prometheus scrapes them.

WRITTEN BY Dharshan Kumar K S

Dharshan Kumar is a Research Associate at CloudThat. He has a working knowledge of various cloud platforms such as AWS, Microsoft, ad GCP. He is interested to learn more about AWS's Well-Architected Framework and writes about them.