CareerByteCode
ByteBlogby CareerByteCode

Alertmanager: Route Alerts to Email, Slack and On-Call

What you build: Alertmanager on GKE with routing, grouping, silences and inhibition tested, a case typo that misroutes a page, and a hot-reload fix. Every step tested.

Alertmanager: Route Alerts to Email, Slack and On-Call

✍️ Translate this blog into Hindi, Tamil or Telugu

Prometheus decides when something is wrong. Alertmanager decides who hears about it, how, and how often. Get Alertmanager wrong and you either miss a real outage or wake people up 50 times for one problem. In this lab you run Alertmanager on Kubernetes with three receivers that stand in for a pager, Slack and email, and test the four features that matter: routing, grouping, silences and inhibition. Then you break routing with a one-letter case typo and fix it with a hot reload.

Who it is for: anyone who runs Prometheus alerts, or will. If Prometheus is new to you, start with Prometheus and Grafana on Kubernetes from Scratch. Time: about 1 hour. Cost: a small GKE cluster for about 10 minutes, covered by the free trial credit. Delete it at the end.

Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud and kubectl); Cloud Shell in your browser has the same tools.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: Alertmanager in plain English

Prometheus sends every firing alert to Alertmanager. Alertmanager then:

  • Routes it: follows a tree of rules (routes) and picks a receiver (a team's pager, a Slack channel, an email list) from the alert's labels (severity, team, cluster).
  • Groups it: alerts with the same group_by labels that arrive close together are sent as one message.
  • Silences it: during planned work, matching alerts are kept but nobody is notified.
  • Inhibits it: while a big alert fires (the whole cluster is down), smaller alerts it causes are hidden.
How Alertmanager handles an alert
Alerts come in with labels; Alertmanager routes, groups, silences and inhibits, then notifies one receiver

Words you will use:

  • Label: a key and value on the alert, for example severity="critical". Everything in Alertmanager works on labels.
  • Matcher: a test on a label: severity="critical" (equal), severity=~"warn.*" (regular expression).
  • Receiver: where notifications go. Real ones use email_configs, slack_configs or pagerduty_configs; ours use webhook_configs pointing to tiny services that print what they get.
  • amtool: Alertmanager's command-line tool. It can check the config, test routes, add test alerts and manage silences.
  • Real-life example: Alertmanager is a hospital switchboard. It sends heart attacks to the cardiologist's pager, groups ten calls about one broken lift into one ticket, ignores the fire alarm test announced for 10 AM, and when the power fails, it does not also page about every machine that lost power.

Before you start

  1. A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
  2. One small GKE cluster. We created it once for this batch of Kubernetes labs:
gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --enable-dataplane-v2

Part B: the lab

Step 1: the configuration

Why: the whole behaviour is in one file. Read it top to bottom: the default receiver is team-email; group_by puts alerts with the same name and cluster together; group_wait: 10s waits 10 seconds to collect a group before sending; two routes send critical alerts to the pager and database alerts to Slack; one inhibit rule hides warnings while ClusterDown fires for the same cluster. Save it as alertmanager.yml:

# Alertmanager configuration: who gets which alert, how alerts are grouped, and which alerts hide others
route:
  receiver: team-email                 # default: anything that matches no route below
  group_by: [alertname, cluster]
  group_wait: 10s                      # wait 10 s to collect alerts of the same group into one message
  group_interval: 1m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      receiver: oncall-pager
    - matchers: [team="db"]
      receiver: db-slack
inhibit_rules:
  - source_matchers: [alertname="ClusterDown"]
    target_matchers: [severity="warning"]
    equal: [cluster]                   # while a cluster is down, hide its warnings
receivers:
  # In this lab every receiver is a small webhook service that prints what it gets.
  # Real teams use email_configs, slack_configs or pagerduty_configs here (see the post).
  - name: oncall-pager
    webhook_configs: [{url: "http://oncall.alerts.svc:8080/"}]
  - name: db-slack
    webhook_configs: [{url: "http://slack.alerts.svc:8080/"}]
  - name: team-email
    webhook_configs: [{url: "http://email.alerts.svc:8080/"}]

The receivers are a 15-line Python web server that prints one line per notification (receiver.py):

# receiver.py: a tiny webhook that stands in for a pager, Slack or email. It prints one line per notification.
import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class H(BaseHTTPRequestHandler):
    def do_POST(self):
        d = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        names = sorted({a["labels"].get("alertname", "?") + "/" + a["labels"].get("instance", "-") for a in d["alerts"]})
        print(f"notification: status={d['status']} alerts={len(d['alerts'])} group={d['groupLabels']} -> {', '.join(names)}", flush=True)
        self.send_response(200); self.end_headers()

    def log_message(self, *a):
        pass


HTTPServer(("0.0.0.0", 8080), H).serve_forever()

And Alertmanager v0.27.0 itself, with its config from a ConfigMap (stack.yaml):

# Alertmanager (config from a ConfigMap) and three webhook receivers that stand in for a pager, Slack and email
apiVersion: apps/v1
kind: Deployment
metadata: {name: alertmanager, namespace: alerts}
spec:
  replicas: 1
  selector: {matchLabels: {app: alertmanager}}
  template:
    metadata: {labels: {app: alertmanager}}
    spec:
      containers:
      - name: alertmanager
        image: prom/alertmanager:v0.27.0
        args: ["--config.file=/etc/am/alertmanager.yml", "--storage.path=/tmp/am"]
        ports: [{containerPort: 9093}]
        volumeMounts: [{name: cfg, mountPath: /etc/am}]
      volumes: [{name: cfg, configMap: {name: am-config}}]
---
apiVersion: v1
kind: Service
metadata: {name: alertmanager, namespace: alerts}
spec: {selector: {app: alertmanager}, ports: [{port: 9093}]}

We create the namespace, the ConfigMaps, the three receivers and Alertmanager:

# Alertmanager plus three receivers (oncall, slack, email): small webhook services that print what they receive
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
kubectl create namespace alerts > /dev/null
kubectl -n alerts create configmap am-config --from-file=alertmanager.yml > /dev/null && kubectl -n alerts create configmap receiver --from-file=receiver.py > /dev/null
for r in oncall slack email; do
  kubectl -n alerts create deployment $r --image=python:3.12-slim -- python /app/receiver.py > /dev/null
  kubectl -n alerts patch deployment $r --type=json -p '[{"op":"add","path":"/spec/template/spec/volumes","value":[{"name":"r","configMap":{"name":"receiver"}}]},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[{"name":"r","mountPath":"/app"}]}]' > /dev/null
  kubectl -n alerts expose deployment $r --port=8080 > /dev/null
done
kubectl apply -f stack.yaml > /dev/null && for d in alertmanager oncall slack email; do kubectl -n alerts rollout status deploy/$d --timeout=120s | tail -1; done
deployment "alertmanager" successfully rolled out
deployment "oncall" successfully rolled out
deployment "slack" successfully rolled out
deployment "email" successfully rolled out

Step 2: check the config and test routes before any alert

Why: amtool check-config catches syntax mistakes, and amtool config routes test tells you which receiver a set of labels would reach, without sending anything. Use both in your CI pipeline.

# Check the config and test the routing tree BEFORE sending anything (amtool ships inside the Alertmanager image)
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
kubectl -n alerts exec deploy/alertmanager -- amtool check-config /etc/am/alertmanager.yml | head -6
for l in "severity=critical" "team=db severity=warning" "team=web severity=warning"; do echo "$l -> $(kubectl -n alerts exec deploy/alertmanager -- amtool config routes test --config.file=/etc/am/alertmanager.yml $l)"; done
Checking '/etc/am/alertmanager.yml'  SUCCESS
Found:
 - global config
 - route
 - 1 inhibit rules
 - 3 receivers
severity=critical -> oncall-pager
team=db severity=warning -> db-slack
team=web severity=warning -> team-email
Config check
Step 1 and 2: four deployments ready, the config checked, three route tests

Step 3: routing, three alerts, three receivers

# Send three alerts. Each must reach a different receiver
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
$AM alert add alertname=ApiDown severity=critical cluster=prod instance=api-1
$AM alert add alertname=SlowQueries severity=warning team=db cluster=prod instance=db-1
$AM alert add alertname=HighMemory severity=warning team=web cluster=prod instance=web-1
sleep 20; for r in oncall slack email; do echo "[$r] $(kubectl -n alerts logs deploy/$r | tail -1)"; done
[oncall] notification: status=firing alerts=1 group={'alertname': 'ApiDown', 'cluster': 'prod'} -> ApiDown/api-1
[slack] notification: status=firing alerts=1 group={'alertname': 'SlowQueries', 'cluster': 'prod'} -> SlowQueries/db-1
[email] notification: status=firing alerts=1 group={'alertname': 'HighMemory', 'cluster': 'prod'} -> HighMemory/web-1

Each alert reached exactly the receiver the tree promised.

Grouping: five alerts, one page

Five nodes go NotReady within seconds. Without grouping, the on-call engineer gets five pages for one problem:

# Grouping: five alerts of the same kind arrive within seconds. The on-call engineer gets ONE notification, not five
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
for i in 1 2 3 4 5; do $AM alert add alertname=NodeNotReady severity=critical cluster=prod instance=node-$i; done
sleep 25; kubectl -n alerts logs deploy/oncall | grep NodeNotReady | tail -1
echo "NodeNotReady notifications: $(kubectl -n alerts logs deploy/oncall | grep -c NodeNotReady)"
notification: status=firing alerts=5 group={'alertname': 'NodeNotReady', 'cluster': 'prod'} -> NodeNotReady/node-1, NodeNotReady/node-2, NodeNotReady/node-3, NodeNotReady/node-4, NodeNotReady/node-5
NodeNotReady notifications: 1

One notification with all five alerts inside (alerts=5), because they share alertname and cluster.

Silences: planned maintenance

Why: during maintenance, alerts are expected. A silence matches labels for a time window:

# Silence: planned maintenance on db-1 for 30 minutes. The alert still exists, but nobody is notified
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
$AM silence add instance=db-1 --duration=30m --comment="planned maintenance" --author=lab > /dev/null && $AM silence query | cut -c1-110
B=$(kubectl -n alerts logs deploy/slack | grep -c DiskFull); $AM alert add alertname=DiskFull severity=warning team=db cluster=prod instance=db-1; sleep 25
echo "DiskFull notifications to slack: $(( $(kubectl -n alerts logs deploy/slack | grep -c DiskFull) - B ))"
$AM alert query alertname=DiskFull --silenced | tail -1
ID                                    Matchers         Ends At                  Created By  Comment           
3409d9e1-b2b6-49b7-a4ba-8c5c8dfa5867  instance="db-1"  2026-11-19 07:56:02 UTC  lab         planned maintenanc
DiskFull notifications to slack: 0
DiskFull   2026-11-19 07:26:03 UTC           suppressed

The DiskFull alert exists (Alertmanager shows it as suppressed), but Slack got 0 notifications.

Inhibition: show the cause, hide the symptoms

The whole staging cluster is down, and a latency warning from the same cluster arrives:

# Inhibition: the whole staging cluster is down. Its warnings are hidden, so on-call sees the cause, not 20 symptoms
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
$AM alert add alertname=ClusterDown severity=critical cluster=staging instance=staging
$AM alert add alertname=HighLatency severity=warning team=web cluster=staging instance=web-9
sleep 25; echo "[oncall] $(kubectl -n alerts logs deploy/oncall | grep ClusterDown | tail -1)"
echo "HighLatency notifications to email: $(kubectl -n alerts logs deploy/email | grep -c HighLatency)"
$AM alert query alertname=HighLatency --inhibited | tail -1
[oncall] notification: status=firing alerts=1 group={'alertname': 'ClusterDown', 'cluster': 'staging'} -> ClusterDown/staging
HighLatency notifications to email: 0
HighLatency  2026-11-19 07:26:40 UTC           suppressed

The pager got ClusterDown; the email receiver got 0 HighLatency notifications, because the inhibit rule hides warnings with the same cluster label.

Routing, grouping, silences, inhibition
Three alerts to three receivers, five alerts in one notification, a silenced DiskFull and an inhibited HighLatency

Break it: a capital letter

A team's exporter sends severity="Critical" with a capital C. Matchers are case sensitive, so the alert misses the pager route and falls through to the default:

📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
# Break it: an alert labelled severity="Critical" (capital C). Matchers are case sensitive: it misses the pager route
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
echo "severity=Critical -> $(kubectl -n alerts exec deploy/alertmanager -- amtool config routes test --config.file=/etc/am/alertmanager.yml severity=Critical)"
$AM alert add alertname=PaymentsDown severity=Critical cluster=prod instance=pay-1; sleep 20
echo "[oncall] PaymentsDown: $(kubectl -n alerts logs deploy/oncall | grep -c PaymentsDown)   [email] PaymentsDown: $(kubectl -n alerts logs deploy/email | grep -c PaymentsDown)"
severity=Critical -> team-email
[oncall] PaymentsDown: 0   [email] PaymentsDown: 1

A payments outage went to an email list instead of the on-call pager. Nothing failed; the routing just did not match.

Fix it: a case-insensitive matcher and a hot reload

We change the matcher to a regular expression, severity=~"(?i)critical" ((?i) means "ignore case"). Two things we learned on the way:

  • Quote the whole matcher. Our first try wrote [severity=~"(?i)critical"] without outer quotes, and that is not valid YAML inside [ ] (Alertmanager refused to load it). Writing ['severity=~"(?i)critical"'] is valid, so the step checks the new file with a YAML parser before using it.
  • Reload, do not restart. A changed ConfigMap reaches the pod after up to a minute or so (each command terminated with exit code 1 line below is one check that did not see the new file yet). Then POST /-/reload tells Alertmanager to read it, with no restart and no lost alerts.
# Fix it: match severity without caring about case (a regex matcher). Update the ConfigMap, wait until the pod sees
# the new file (up to a minute), then tell Alertmanager to reload it: no restart, no lost alerts
kubectl -n alerts get configmap am-config -o jsonpath='{.data.alertmanager\.yml}' | sed 's/matchers: \[severity="critical"\]/matchers: ['"'"'severity=~"(?i)critical"'"'"']/' > fixed.yml && grep "severity=~" fixed.yml
python3 -c "import yaml; yaml.safe_load(open('fixed.yml'))" && echo "fixed.yml is valid YAML"
kubectl -n alerts create configmap am-config --from-file=alertmanager.yml=fixed.yml --dry-run=client -o yaml | kubectl replace -f - > /dev/null
for i in $(seq 1 30); do kubectl -n alerts exec deploy/alertmanager -- grep -q "severity=~" /etc/am/alertmanager.yml && break; sleep 5; done; echo "new file in the pod after about $((i*5)) s"
kubectl -n alerts exec deploy/alertmanager -- wget -qO- --post-data="" http://localhost:9093/-/reload && echo "reloaded"
echo "severity=Critical -> $(kubectl -n alerts exec deploy/alertmanager -- amtool config routes test --config.file=/etc/am/alertmanager.yml severity=Critical)"
kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093 alert add alertname=CheckoutDown severity=Critical cluster=prod instance=pay-2; sleep 20
echo "[oncall] CheckoutDown: $(kubectl -n alerts logs deploy/oncall | grep -c CheckoutDown)   [email] CheckoutDown: $(kubectl -n alerts logs deploy/email | grep -c CheckoutDown)"
    - matchers: ['severity=~"(?i)critical"']
fixed.yml is valid YAML
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
new file in the pod after about 70 s
reloaded
severity=Critical -> oncall-pager
[oncall] CheckoutDown: 1   [email] CheckoutDown: 0

Now severity=Critical routes to oncall-pager, and a new CheckoutDown alert with a capital C reached the pager, not email.

Break it and fix it
Critical with a capital C went to email; after the regex matcher and a reload it goes to the pager

POC: prove it works

  • The config passed amtool check-config, and route tests matched the tree (Step 2).
  • Three alerts reached three different receivers (Step 3).
  • Five alerts became one notification; a silence and an inhibit rule each stopped a notification (Grouping, Silences, Inhibition).
  • A case typo misrouted an alert; a quoted regex matcher and a hot reload fixed it (Break it, Fix it).

From webhooks to real receivers

Replace a webhook_configs entry with the real thing. These are the shapes from the Alertmanager documentation; we did not send real email, Slack or PagerDuty messages in this lab:

receivers:
  - name: team-email
    email_configs: [{to: "team@example.com"}]           # needs global smtp_smarthost and smtp_from
  - name: db-slack
    slack_configs: [{channel: "#db-alerts", api_url_file: /etc/am/secrets/slack-url}]
  - name: oncall-pager
    pagerduty_configs: [{routing_key_file: /etc/am/secrets/pd-key}]

Keep webhook URLs and keys in a Kubernetes Secret mounted as files (*_file settings), never in the ConfigMap.

Clean up (delete everything)

# Clean up: delete the namespace (Alertmanager, receivers, config)
kubectl delete namespace alerts --wait=true 2>&1 | tail -1
rm -f fixed.yml
namespace "alerts" deleted

When you have finished all your Kubernetes labs, delete the cluster:

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet

FAQ

Where do the alerts come from in real life? From Prometheus alert rules (PrometheusRule objects with the Prometheus Operator). Here we added them with amtool alert add so you can test Alertmanager on its own.

**What do group_wait, group_interval and repeat_interval do?** Wait before the first message of a new group (to collect more alerts); wait before sending an update when the group changes; resend a still-firing group after this long.

Silence or inhibit? A silence is manual and time-limited (maintenance). An inhibit rule is automatic and always on (cause hides symptoms).

How do I avoid case problems for good? Agree on label values in your alert rules, test routes with amtool config routes test in CI, and use =~"(?i)..." where you cannot control the sender.

Is one Alertmanager enough? For production, run 3 replicas in a cluster (--cluster.peer) so notifications are not lost or duplicated. This lab runs one.

Troubleshooting

You seeWhyFix
did not find expected ',' or ']'A matcher with special characters inside [ ] without quotesQuote each matcher: ['severity=~"(?i)critical"']
An alert reaches the default receiverNo route matched (case, typo, missing label)amtool config routes test <labels>
Ten pages for one problemgroup_by too narrowGroup by alertname and cluster (or service)
A changed ConfigMap does nothingThe pod sees the file only after a delay, and Alertmanager reads it only on reloadWait for the file, then POST /-/reload
A silence does not workThe matcher does not match the alert's labels exactlyamtool silence query and compare labels

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Observability Interview Questions with Hands-On Proof⏱ 17 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →