Prometheus Alerting: Telegram, Email, PagerDuty with AlertManager

When IT infrastructure scales, monitoring becomes essential, but managing alerts can quickly turn into a nightmare. I’ve seen situations where 300 alerts were generated in a single night of scheduled maintenance. The entire team read them, but none were actually relevant. The result? The team started ignoring notifications. The following week, a critical real alert was ignored for four hours, causing significant downtime. The problem wasn’t a lack of monitoring, but a complete absence of intelligent routing and effective silences.

Prometheus is a de facto standard for time-series monitoring. AlertManager, its complementary component, is designed to manage and dispatch alerts generated by Prometheus, ensuring the right notifications reach the right people, through the right channel, at the right time. Strategically configuring AlertManager is fundamental to transforming a chaotic stream of notifications into an effective alerting system that reduces noise and focuses attention on real problems. This article will guide you through configuring AlertManager to send notifications to Telegram, email, and PagerDuty, implementing advanced routing, inhibitions, and silences for an enterprise environment with hundreds of VMs.

Prerequisites / Test Environment

To follow this guide, you will need a working Prometheus installation already collecting metrics. The ideal test environment includes a Linux server (e.g., Ubuntu 24.04 LTS) with Docker or a direct installation of Prometheus and AlertManager. Ensure you have access to a Telegram account (to create a bot), an SMTP server for sending emails, and a PagerDuty account (for critical notifications). For our examples, we will use node_exporter to generate system metrics.

AlertManager Architecture: Receivers, Routes, Inhibitions, Silences

AlertManager does more than just forward alerts. Its architecture is designed to process alerts intelligently. The key components are:

• Receivers: Define where and how notifications will be sent (e.g., Telegram, email, PagerDuty, Slack, Webhook). Each receiver has a unique name.
• Routes: These are the rules that determine which alerts are sent to which receivers. You can define hierarchies and conditions based on alert labels (e.g., severity, team, instance). Routes are organized in a tree structure, with a default_route that catches all alerts not handled by other rules.
• Inhibitions: Prevent redundant or related notifications from being sent. For example, if a server is offline, AlertManager can inhibit all alerts related to services running on that server, avoiding an explosion of notifications and focusing attention on the primary alert (the offline server).
• Silences: Allow temporary disabling of notifications for a specific set of alerts, useful during scheduled maintenance or incident investigation. They can be created via the AlertManager web interface or using the amtool CLI.

Understanding these concepts is crucial for building a robust and non-intrusive alerting system. A well-configured system reduces “noise” by 70%, as I’ve observed in an environment with 2,000 endpoints and 300+ VMs, improving team responsiveness by 50%.

Install and Configure AlertManager

Installing AlertManager is straightforward. You can download it as a standalone binary or run it via Docker. For a production environment, I recommend using Docker or integrating it into an orchestrator like Kubernetes.

Docker Installation (example):

Let’s create an alertmanager.yml file for configuration:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'default-receiver'

receivers:
  - name: 'default-receiver'
    email_configs:
      - to: '[email protected]'
        send_resolved: true

This is a minimal example. Now let’s start AlertManager:

docker run -d \n  -p 9093:9093 \n  -v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml \n  --name alertmanager \n  prom/alertmanager

Access the AlertManager web interface at http://localhost:9093.

Telegram Receiver: Real-time Mobile Notifications

Telegram is an excellent channel for warning alerts or notifications that require immediate attention but are not critical. The first step is to create a Telegram bot and obtain the bot_token and chat_id.

Create a Telegram bot:

1. Search for @BotFather on Telegram and start a conversation.
2. Use the /newbot command and follow the instructions to name your bot. You will be provided with the bot_token.
3. Start a conversation with your new bot. Then, to get the chat_id, visit https://api.telegram.org/bot/getUpdates in your browser. Send a message to the bot, then refresh the page. You will find the chat_id in the JSON response.

Configuration in alertmanager.yml:

receivers:
  - name: 'telegram-warnings'
    telegram_configs:
      - bot_token: '<YOUR_TELEGRAM_BOT_TOKEN>'
        chat_id: <YOUR_TELEGRAM_CHAT_ID>
        message: '{{ .CommonAnnotations.summary }} - Severity: {{ .CommonLabels.severity }}'
        parse_mode: 'HTML'
        send_resolved: true

Replace and with the values you obtained.

Email Receiver with SMTP

For more formal notifications or alerts requiring documented action, email is a reliable channel. AlertManager supports sending via an SMTP server.

Configuration in alertmanager.yml:

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: 'smtpuser'
  smtp_auth_password: 'smtppassword'
  smtp_require_tls: true

receivers:
  - name: 'email-criticals'
    email_configs:
      - to: '[email protected]'
        subject: 'CRITICAL ALERT: {{ .CommonLabels.alertname }}'
        send_resolved: true

Ensure that smtp_smarthost, smtp_from, smtp_auth_username, and smtp_auth_password parameters are correct for your SMTP server. For security reasons, it is advisable to use environment variables or a Vault for credentials.

Routing Rules: Send Critical Alerts to PagerDuty, Warnings to Telegram

Here lies the true power of AlertManager: defining precise rules for alert forwarding. We want critical alerts to go to PagerDuty, while warning alerts go to Telegram.

Configuration in alertmanager.yml:

route:
  group_by: ['alertname', 'instance', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default-email'
  routes:
    - match:
        severity: 'critical'
      receiver: 'pagerduty-criticals'
    - match:
        severity: 'warning'
      receiver: 'telegram-warnings'

receivers:
  - name: 'default-email'
    email_configs:
      - to: '[email protected]'

  - name: 'telegram-warnings'
    telegram_configs:
      - bot_token: '<YOUR_TELEGRAM_BOT_TOKEN>'
        chat_id: <YOUR_TELEGRAM_CHAT_ID>

  - name: 'pagerduty-criticals'
    pagerduty_configs:
      - service_key: '<YOUR_PAGERDUTY_SERVICE_KEY>'
        send_resolved: true

In this configuration, default-email will receive all alerts that do not match the critical or warning rules. Alerts with severity: critical will be sent to PagerDuty (you’ll need to generate a service_key from the PagerDuty interface), while those with severity: warning will go to Telegram. This approach ensures that only alerts requiring immediate action interrupt the on-call team’s sleep.

Inhibitions: Avoid Derived Alerts During an Incident

Inhibitions are crucial for preventing alert fatigue. Imagine a server hosting 10 different services. If the server goes offline, you would receive 10 alerts for the services and 1 for the server. With inhibitions, you can configure AlertManager to send only the server offline alert and suppress the others.

Configuration in alertmanager.yml (inhibit_rules section):

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    # Alerts with 'critical' severity inhibit those with 'warning'
    # if they share the 'instance' and 'job' labels
    equal: ['instance', 'job']

  - source_match:
      alertname: 'HostDown'
    target_match:
      alertname: 'ServiceDown'
    equal: ['instance']

The first rule inhibits warning alerts if a critical alert with the same instance and job already exists. The second inhibits ServiceDown alerts if HostDown is already active for the same instance. This reduces noise and helps the team focus on the root cause.

Silences: Pause Alerts During Maintenance

During scheduled maintenance windows (e.g., software updates, server reboots), alerts are typically generated. Silences allow you to temporarily suppress these notifications.

You can create a silence via the AlertManager web interface or using the amtool CLI:

amtool silence add alertname=DiskWarning instance=my-server-01 --duration=2h --comment="Scheduled disk maintenance" --expire-on-resolve

This command creates a silence for the DiskWarning alert on instance my-server-01 for two hours. The --expire-on-resolve flag ensures the silence is removed as soon as the alert resolves or expires.

Practical Alert Rule Examples: Disk, CPU, Memory, Services

Alert rules are defined in the Prometheus configuration file (e.g., prometheus.yml or in separate files included via rule_files). Here are some common examples:

Alert for nearly full disk space:

# rules.yml

groups:
  - name: 'host-alerts'
    rules:
      - alert: DiskWarning
        expr: (node_filesystem_avail_bytes{fstype!=""} / node_filesystem_size_bytes{fstype!=""}) * 100 < 15
        for: 5m
        labels:
          severity: 'warning'
        annotations:
          summary: 'Disk nearly full on {{ $labels.instance }} ({{ $value | printf "%.2f" }}%)'
          description: 'Available disk space on {{ $labels.device }} of {{ $labels.instance }} is below 15%.'

      - alert: DiskCritical
        expr: (node_filesystem_avail_bytes{fstype!=""} / node_filesystem_size_bytes{fstype!=""}) * 100 < 5
        for: 2m
        labels:
          severity: 'critical'
        annotations:
          summary: 'CRITICAL: Disk full on {{ $labels.instance }} ({{ $value | printf "%.2f" }}%)'
          description: 'Available disk space on {{ $labels.device }} of {{ $labels.instance }} is below 5%. Act immediately!'

Alert for high CPU usage:

      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: 'critical'
        annotations:
          summary: 'High CPU usage on {{ $labels.instance }} ({{ $value | printf "%.2f" }}%)'
          description: 'CPU on {{ $labels.instance }} is over 90% utilized in the last 5 minutes.'

Alert for insufficient RAM:

      - alert: InsufficientMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 5m
        labels:
          severity: 'warning'
        annotations:
          summary: 'Insufficient memory on {{ $labels.instance }} ({{ $value | printf "%.2f" }}%)'
          description: 'Available memory on {{ $labels.instance }} is below 10%.'

These examples are a good starting point. Remember to adapt them to your specific thresholds and needs. For a complete reference, consult the official Prometheus Alerting documentation.

Common Errors and Troubleshooting

• YAML configuration error: Often, the problem is incorrect YAML formatting. Use a YAML linter to validate your alertmanager.yml file.
• Incorrect credentials: Invalid Telegram bot_token, PagerDuty service_key, or SMTP credentials. Check AlertManager logs (docker logs alertmanager).
• Missing labels: Routing and inhibition rules rely on labels. If an alert does not have the expected labels (e.g., severity), it will not be routed correctly. Verify alert labels in Prometheus.
• Firewall: Ensure AlertManager can reach external endpoints (Telegram API, SMTP server, PagerDuty API). Check firewall rules.
• amtool check-config: Use this command to validate the syntax of your configuration file before reloading or restarting AlertManager.

FAQ — Frequently Asked Questions

How can I test my AlertManager configurations without generating real alerts?

You can use amtool alert send to send test alerts to AlertManager. This allows you to verify if routing rules and receivers work as expected without waiting for a metric to cross a real threshold. For example: amtool alert send -r 'DiskWarning' -l 'instance=my-server-01,severity=warning'.

What is the difference between group_wait, group_interval, and repeat_interval?

group_wait is the time AlertManager waits after receiving the first alert of a new group before sending the notification. group_interval is the time AlertManager waits between sending notifications for the same alert group. repeat_interval is the minimum time between sending the same notification for a persistent alert, even if there are no new alerts in the same group.

Can I integrate AlertManager with ticketing systems like Jira?

Yes, AlertManager can be integrated with ticketing systems via webhooks. You can configure a webhook_configs receiver and point it to an API that handles creating or updating tickets in Jira. There are also third-party integrations or custom scripts that facilitate this process.

How do I securely manage sensitive credentials in alertmanager.yml?

You should never embed sensitive credentials directly in the configuration file. Use environment variables (e.g., $${ENV_VAR_NAME}) that AlertManager can read, or a secret management system like HashiCorp Vault or Kubernetes Secrets, injecting credentials as environment variables into the AlertManager container.

Conclusions with Operational Takeaways

Configuring AlertManager is a critical step for any infrastructure relying on Prometheus. By implementing intelligent routing, inhibitions, and silences, you can transform a noisy and ineffective alerting system into a powerful tool that notifies your team only of the issues that matter. Remember to always test your configurations and periodically review your alert rules to ensure they remain relevant. A well-oiled alerting system is the first line of defense against downtime and ensures your team can react proactively, not reactively.

Updated: July 2026

Read also

Read also: Prometheus Grafana Linux: Server Monitoring Guide 2026

Read also: CI/CD Pipeline Sicura: Checklist 12 Punti per GitLab e GitHub Actions (2026)

Read also: Oracle DBA Checklist: 10 Daily Checks for Production Databases (2026)