Maximizing Monitoring Capabilities with Telegraf

Overview

Telegraf is a powerful open-source server agent written in Go designed to collect, process, and send metrics and data to various outputs. Its modular architecture makes it incredibly versatile, allowing users to easily configure it by adding different plugins based on their specific monitoring needs. Whether you’re monitoring system performance, databases, or applications, Telegraf offers various plugins to gather data from diverse sources.

Telegraf and Its Modularity

At its core, Telegraf operates on the concept of plugins, which act as building blocks for data collection and transmission. These plugins can be categorized into Input, Processor, Aggregator, and Output plugins, each serving a distinct purpose in the data pipeline. The plugins together are kept in a configuration file. A default config file can be generated in telegraf by the command:

telegraf config > telegraf.conf

1	telegraf config > telegraf.conf

Input Plugins

Input plugins are responsible for gathering data from various sources. Some of the most commonly used input plugins include:

CPU Input Plugin: It monitors CPU usage metrics, such as usage percentage and load average.

[[inputs.cpu]]
  percpu = true
  totalcpu = true

[[inputs.cpu]]

percpu = true

totalcpu = true

Disk Input Plugin: Collects disk metrics including usage, I/O details, and disk space.

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs"]

1 2	[[inputs.disk]] ignore_fs = ["tmpfs", "devtmpfs", "devfs"]

HTTP Input Plugin: Fetches data from HTTP endpoints, which is useful for retrieving information from APIs.

[[inputs.http]]
  urls = ["http://example.com/metrics"]

1 2	[[inputs.http]] urls = ["http://example.com/metrics"]

Kafka Input Plugin: Telegraf’s Kafka input plugin allows seamless integration to consume data from Kafka topics.

[[inputs.kafka_consumer]]
  brokers = ["kafka-server1:9092", "kafka-server2:9092"]
  topics = ["topic1", "topic2"]
  group_id = "telegraf_consumers"
  data_format = "json"

[[inputs.kafka_consumer]]

brokers = ["kafka-server1:9092", "kafka-server2:9092"]

topics = ["topic1", "topic2"]

group_id = "telegraf_consumers"

data_format = "json"

This configuration subscribes to topics “topic1” and “topic2,” consuming JSON-formatted messages from Kafka brokers running on servers kafka-server1 and kafka-server2 on port 9092.

MQTT Input Plugin: Telegraf’s MQTT input plugin facilitates data ingestion from MQTT brokers.

[[inputs.mqtt_consumer]]
  servers = ["tcp://mqtt-broker1:1883", "tcp://mqtt-broker2:1883"]
  topics = ["sensor/data"]
  qos = 0
  client_id = "telegraf_client"
  data_format = "json"

[[inputs.mqtt_consumer]]

servers = ["tcp://mqtt-broker1:1883", "tcp://mqtt-broker2:1883"]

topics = ["sensor/data"]

qos = 0

client_id = "telegraf_client"

data_format = "json"

This setup subscribes to the “sensor/data” topic on MQTT brokers running on mqtt-broker1 and mqtt-broker2 via port 1883, consuming JSON-formatted sensor data.

Processor Plugins

Processor plugins manipulate, filter, or modify incoming data before passing it to the next stage. Examples include:

Grok Processor Plugin: Parses unstructured log data into structured formats using regex patterns.

[[processors.grok]]
  patterns = ["%{COMBINED_LOG_FORMAT}"]
  measurement = "logs"

[[processors.grok]]

patterns = ["%{COMBINED_LOG_FORMAT}"]

measurement = "logs"

Converter Processor Plugin: Converts field types (e.g., integer to float) for compatibility with outputs.

[[processors.converter]]
  [processors.converter.fields]
    integer_field = ["float"]

[[processors.converter]]

[processors.converter.fields]

integer_field = ["float"]

Aggregator Plugins

Aggregator plugins summarize or aggregate data over a defined period before sending it out. Examples include:

Statsd Aggregator Plugin: Aggregates metrics for sending to a StatsD server.

[[aggregators.statsd]]
  service_address = ":8125"

1 2	[[aggregators.statsd]] service_address = ":8125"

BasicStats Aggregator Plugin: Computes basic statistical measurements (mean, median, min, max) for collected metrics.

[[aggregators.basicstats]]
  period = "30s"
  drop_original = true

[[aggregators.basicstats]]

period = "30s"

drop_original = true

Histogram Aggregator Plugin: Collects and computes histogram distribution of input metrics.

[[aggregators.histogram]]
  buckets = [0.1, 1, 5, 10, 50, 100, 500]

1 2	[[aggregators.histogram]] buckets = [0.1, 1, 5, 10, 50, 100, 500]

Output Plugins

Output plugins transmit the collected and processed data to various destinations. A few examples include:

InfluxDB Output Plugin: Sends metrics to an InfluxDB time series database.

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "mydb"

[[outputs.influxdb]]

urls = ["http://localhost:8086"]

database = "mydb"

Prometheus Output Plugin: Pushes metrics to a Prometheus server for monitoring.

[[outputs.prometheus_client]]
  listen = ":9273"

1 2	[[outputs.prometheus_client]] listen = ":9273"

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Some Real-World Examples

Let’s consider a scenario where Telegraf is used to monitor a web server and a database:

Web Server Monitoring

Input Plugins:

CPU Input: Collects CPU usage data.

Disk Input: Gathers disk metrics.

Processor Plugins:

Grok Processor: Parses access logs into structured data.

Output Plugins:

InfluxDB Output: Sends processed data to an InfluxDB instance for storage.

Database Monitoring

Input Plugins

CPU Input: Monitors CPU performance.

Disk Input: Tracks disk usage metrics.

MySQL Input: Collects database-specific metrics for MySQL.

Processor Plugins

Converter Processor: Converts field types if needed.

Output Plugins

Prometheus Output: Pushes aggregated data to a Prometheus server for monitoring.

Another efficient solution may be writing MQTT data to an InfluxDB or Kafka. A sample config file is shown below.

[[inputs.mqtt_consumer]]
  servers = ["tcp://mqtt-broker1:1883", "tcp://mqtt-broker2:1883"]
  topics = ["sensor/data"]
  qos = 0
  client_id = "telegraf_client"
  data_format = "json"

[[processors.converter]]
  [processors.converter.fields]
    integer_field = ["float"]

[[outputs.kafka]]
  brokers = ["kafka-server1:9092"]
  topic = "processed_data"

[[outputs.influxdb]]
  urls = ["http://influxdb-server:8086"]
  database = "iot_data"

[[inputs.mqtt_consumer]]

servers = ["tcp://mqtt-broker1:1883", "tcp://mqtt-broker2:1883"]

topics = ["sensor/data"]

qos = 0

client_id = "telegraf_client"

data_format = "json"

[[processors.converter]]

[processors.converter.fields]

integer_field = ["float"]

[[outputs.kafka]]

brokers = ["kafka-server1:9092"]

topic = "processed_data"

[[outputs.influxdb]]

urls = ["http://influxdb-server:8086"]

database = "iot_data"

Conclusion

Telegraf’s flexibility and extensive plugin ecosystem make it a go-to solution for collecting and forwarding metrics from diverse sources. By leveraging its modular architecture, users can tailor their monitoring setups to suit specific requirements, whether for system health checks, application performance monitoring, or database analysis.

With Telegraf’s rich assortment of plugins, the possibilities for monitoring and analyzing data are endless.

Drop a query if you have any questions regarding Telegraf and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. What is Telegraf, and how does it enhance monitoring capabilities?

ANS: – Telegraf is a versatile server agent using dynamic plugins, empowering diverse data collection and transmission for enhanced monitoring capabilities.

2. Can Telegraf accommodate complex monitoring setups involving multiple data sources?

ANS: – Telegraf supports complex monitoring setups by enabling multiple inputs, processors, and aggregators for diverse data sources.

WRITTEN BY Rishi Raj Saikia

Rishi Raj Saikia is working as Sr. Research Associate - Data & AI IoT team at CloudThat. He is a seasoned Electronics & Instrumentation engineer with a history of working in Telecom and the petroleum industry. He also possesses a deep knowledge of electronics, control theory/controller designing, and embedded systems, with PCB designing skills for relevant domains. He is keen on learning new advancements in IoT devices, IIoT technologies, and cloud-based technologies.