Voiced by Amazon Polly |
Introduction
A solid monitoring and alerting system is crucial for keeping your applications running smoothly in modern cloud-native environments. At the heart of the Prometheus ecosystem sits Alertmanager, which deduplicates, groups, and routes alerts so that the right people see them fast, whether through Microsoft Teams, email, or another channel.
When a metric, log, or trace (from Prometheus, Loki, Tempo, etc.) crosses a defined threshold, an alert fires in Prometheus and lands in Alertmanager. From there, you decide how to deliver it: send it straight to Teams or email, or slip it through an adapter like prometheus‑msteams for extra formatting power.
You have three main paths for getting Prometheus alerts in front of your team:
- Prometheus -> Alertmanager -> Prometheus‑msteams -> Microsoft Teams
- Prometheus → Alertmanager → Microsoft Teams
- Prometheus → Alertmanager → Email
Pioneers in Cloud Consulting & Migration Services
- Reduced infrastructural costs
- Accelerated application deployment
Basic Setup: Prometheus Alert Rules & Alertmanager Configuration
Set up the core components Prometheus and Alertmanager, which will be common across all three approaches. To enable alerting, you must define threshold-based alert rules in Prometheus and connect it to Alertmanager using values.yaml. Prometheus pushes alerts to Alertmanager when a condition is met.
- Prometheus Alert Rules
Prometheus evaluates conditions using alert rules. These rules define expressions that trigger alerts when certain failure states or thresholds are met. Below is an example to detect when a pod enters a failure state like CrashLoopBackOff, ImagePullBackOff
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
serverFiles: alerting_rules.yml: groups: - name: pod-failure-alerts rules: - alert: PodInFailureState expr: kube_pod_container_status_waiting_reason{reason=~"CrashLoopBackOff|ImagePullBackOff|ErrImagePull|Error"} > 0 for: 2m labels: severity: critical annotations: summary: "Pod failure detected ({{ $labels.namespace }}/{{ $labels.pod }})" description: | Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is in a failure state: {{ $labels.reason }}. |
These rules can be placed in a file like alert-rules.yaml and mounted using Helm.
- Connecting Prometheus to Alertmanager
To forward alerts to Alertmanager, Prometheus must be configured with the Alertmanager service endpoint. This is done in values.yaml:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
serverFiles: prometheus.yml: rule_files: - /etc/config/recording_rules.yml - /etc/config/alerting_rules.yml - /etc/config/rules - /etc/config/alerts alerting: alertmanagers: - static_configs: - targets: - 'alertmanager.{{.Release.Namespace}}.svc.cluster.local:9093' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt |
Make sure the Alertmanager service is accessible within the Kubernetes cluster at alertmanager:9093.
- Alertmanager Basic Routing Config
Set up values.yaml for Alertmanager to route incoming alerts. Start with a basic receiver and customize later based on the method (MS Teams, email, etc.):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
config: enabled: true global: resolve_timeout: 5m templates: - '/etc/alertmanager/*.tmpl' route: group_by: ['alertname', 'namespace', 'pod'] group_wait: 5s group_interval: 5m receiver: 'default_receiver' repeat_interval: 1h routes: - receiver: 'default_receiver' # matchers: # - 'severity=warning' # uncomment below line to enable email alerts # - receiver: 'email' # matchers: # - 'severity=critical' receivers: - name: 'null' - name: 'default_receiver' email_configs: [] webhook_configs: [] |
This connects Prometheus to Alertmanager and sets up basic alert delivery. You can extend it for MS Teams, email, or other notifiers.
Method 1: Using Prometheus-msteams
Previous versions of Alertmanager couldn’t send alerts straight to Microsoft Teams because they didn’t support Adaptive Cards. To work around this, we use the prometheus-msteams adapter as a bridge: it takes Alertmanager’s alerts, converts them into Teams-friendly Adaptive Cards, and delivers them to your chosen channel. Plus, it lets you fully customize your message templates to tailor alerts exactly how you like.
Webhook-URL Setup in Teams: –
- Create a Team and a channel inside.
- Go to Channel’s Manage Channel à click on Connectors Edit à Select Incoming Webhook’s Configure à provide name, and click on generate. Copy the Webhook URL.
Configuration Setup:
1 2 3 4 5 6 7 8 |
receivers: - name: 'null' - name: 'default_receiver' webhook_configs: - url: 'http://prometheus-msteams:2000/alert' send_resolved: true http_config: follow_redirects: true |
Prometheus-msteams configuration: Default message template
1 2 |
connectors: - alert: <webhook_url> |
With customization of alert message: A sample customized message template
Displaying alert message in MS-Teams:
Method 2: Direct MS Teams Webhook
Recent Alertmanager versions (from v0.28.1) support Microsoft Teams via webhook_configs with Adaptive Cards. But message customization is more limited than prometheus‑msteams, but you avoid running an extra component.
Setup of Workflow webhook URL in Teams: –
- Go to Workflow App in Teams. Click on New flow.
- Select Post to a channel when a webhook request is received -> click on Next -> Select Team’s Team and Team’s Channel, and Click on Create flow. Copy the Webhook URL.
Configuration Setup:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
receivers: - name: 'null' - name: 'msteams' msteamsv2_configs: - webhook_url: '<webhook_url' http_config: follow_redirects: true title: '🔥 Alert: {{ .CommonLabels.alertname }}' text: | **Namespace:** {{ .CommonLabels.namespace }} **Pod:** {{ .CommonLabels.pod }} **Status:** {{ .Status }} **Description:** {{ .CommonAnnotations.description }} |
Displaying alert message in MS-Teams:
Method 3: Sending to Email
Alertmanager can send alerts directly via SMTP to email recipients.
Before setting up the configuration, generate app-password. You can get into Google account manager, go to Security, which you can find here, and generate the app-password. You need to enable 2-factor authentication for your Google account to generate app-password.
Configuration Setup:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
email_configs: - to: example@gmail.com from: example@gmail.com # Update your from mail id here smarthost: smtp.gmail.com:587 auth_username: example@gmail.com # Update your from mail id here auth_identity: example@gmail.com # Update your from mail id here auth_password: password # Update your app-password here send_resolved: true headers: subject: " Prometheus - Alert " text: |- 🔔 *Alert:* {{ .CommonLabels.alertname }} Status: {{ .Status | title }} 🔥 Severity: {{ .CommonLabels.severity }} 📍 Namespace: {{ .CommonLabels.namespace }} 🚀 Pod: {{ .CommonLabels.pod }} 🚚 Container: {{ .CommonLabels.container | default "N/A" }} 📌 Description: {{ .CommonAnnotations.description }} |
Replace the ‘example@gmail.com’ with your email address and replace the password with your app-password.
Conclusion
Whether using prometheus-msteams for advanced customization, direct webhooks for simplicity, or email for traditional workflows, each method offers a reliable way to stay informed.
With proper alert rules, Alertmanager routing, and Helm-based deployment, you build a flexible and efficient alerting system that fits your operational needs.
Drop a query if you have any questions regarding Alertmanager and we will get back to you quickly.
Making IT Networks Enterprise-ready – Cloud Management Services
- Accelerated cloud migration
- End-to-end view of the cloud environment
About CloudThat
CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.
FAQs
1. Which method is best for MS Teams integration?
ANS: – If the channel is a stage and you want flashy layouts, prometheus-msteams is your tool. You can use the direct-webhook Adaptive message method for fast setting up configuration, but it has limited customization of message alerts.
2. Can I use multiple receivers in one Alertmanager config?
ANS: – Yes, you can define more receivers and routes and route the alerts based on labels, alert groups, severity, etc.
WRITTEN BY Nallagondla Nikhil
Nallagondla Nikhil works as a Research Intern at CloudThat. He is passionate about continuously expanding his skill set and knowledge base by actively seeking opportunities to learn new skills. Nikhil regularly explores blogs and articles related to various technologies and industry trends to stay up to date with the latest developments in the field.
Comments