Alertmanager: Route Alerts to Email, Slack and On-Call
What you build: Alertmanager on GKE with routing, grouping, silences and inhibition tested, a case typo that misroutes a page, and a hot-reload fix. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
Prometheus decides when something is wrong. Alertmanager decides who hears about it, how, and how often. Get Alertmanager wrong and you either miss a real outage or wake people up 50 times for one problem. In this lab you run Alertmanager on Kubernetes with three receivers that stand in for a pager, Slack and email, and test the four features that matter: routing, grouping, silences and inhibition. Then you break routing with a one-letter case typo and fix it with a hot reload.
Who it is for: anyone who runs Prometheus alerts, or will. If Prometheus is new to you, start with Prometheus and Grafana on Kubernetes from Scratch. Time: about 1 hour. Cost: a small GKE cluster for about 10 minutes, covered by the free trial credit. Delete it at the end.
Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud and kubectl); Cloud Shell in your browser has the same tools.
Part A: Alertmanager in plain English
Prometheus sends every firing alert to Alertmanager. Alertmanager then:
- Routes it: follows a tree of rules (
routes) and picks a receiver (a team's pager, a Slack channel, an email list) from the alert's labels (severity,team,cluster). - Groups it: alerts with the same
group_bylabels that arrive close together are sent as one message. - Silences it: during planned work, matching alerts are kept but nobody is notified.
- Inhibits it: while a big alert fires (the whole cluster is down), smaller alerts it causes are hidden.

Words you will use:
- Label: a key and value on the alert, for example
severity="critical". Everything in Alertmanager works on labels. - Matcher: a test on a label:
severity="critical"(equal),severity=~"warn.*"(regular expression). - Receiver: where notifications go. Real ones use
email_configs,slack_configsorpagerduty_configs; ours usewebhook_configspointing to tiny services that print what they get. - amtool: Alertmanager's command-line tool. It can check the config, test routes, add test alerts and manage silences.
- Real-life example: Alertmanager is a hospital switchboard. It sends heart attacks to the cardiologist's pager, groups ten calls about one broken lift into one ticket, ignores the fire alarm test announced for 10 AM, and when the power fails, it does not also page about every machine that lost power.
Before you start
- A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
- One small GKE cluster. We created it once for this batch of Kubernetes labs:
gcloud container clusters create lab-cluster \
--zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
--disk-type=pd-standard --disk-size=50 --release-channel=regular \
--no-enable-autoupgrade --enable-dataplane-v2Part B: the lab
Step 1: the configuration
Why: the whole behaviour is in one file. Read it top to bottom: the default receiver is team-email; group_by puts alerts with the same name and cluster together; group_wait: 10s waits 10 seconds to collect a group before sending; two routes send critical alerts to the pager and database alerts to Slack; one inhibit rule hides warnings while ClusterDown fires for the same cluster. Save it as alertmanager.yml:
# Alertmanager configuration: who gets which alert, how alerts are grouped, and which alerts hide others
route:
receiver: team-email # default: anything that matches no route below
group_by: [alertname, cluster]
group_wait: 10s # wait 10 s to collect alerts of the same group into one message
group_interval: 1m
repeat_interval: 4h
routes:
- matchers: [severity="critical"]
receiver: oncall-pager
- matchers: [team="db"]
receiver: db-slack
inhibit_rules:
- source_matchers: [alertname="ClusterDown"]
target_matchers: [severity="warning"]
equal: [cluster] # while a cluster is down, hide its warnings
receivers:
# In this lab every receiver is a small webhook service that prints what it gets.
# Real teams use email_configs, slack_configs or pagerduty_configs here (see the post).
- name: oncall-pager
webhook_configs: [{url: "http://oncall.alerts.svc:8080/"}]
- name: db-slack
webhook_configs: [{url: "http://slack.alerts.svc:8080/"}]
- name: team-email
webhook_configs: [{url: "http://email.alerts.svc:8080/"}]The receivers are a 15-line Python web server that prints one line per notification (receiver.py):
# receiver.py: a tiny webhook that stands in for a pager, Slack or email. It prints one line per notification.
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
class H(BaseHTTPRequestHandler):
def do_POST(self):
d = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
names = sorted({a["labels"].get("alertname", "?") + "/" + a["labels"].get("instance", "-") for a in d["alerts"]})
print(f"notification: status={d['status']} alerts={len(d['alerts'])} group={d['groupLabels']} -> {', '.join(names)}", flush=True)
self.send_response(200); self.end_headers()
def log_message(self, *a):
pass
HTTPServer(("0.0.0.0", 8080), H).serve_forever()And Alertmanager v0.27.0 itself, with its config from a ConfigMap (stack.yaml):
# Alertmanager (config from a ConfigMap) and three webhook receivers that stand in for a pager, Slack and email
apiVersion: apps/v1
kind: Deployment
metadata: {name: alertmanager, namespace: alerts}
spec:
replicas: 1
selector: {matchLabels: {app: alertmanager}}
template:
metadata: {labels: {app: alertmanager}}
spec:
containers:
- name: alertmanager
image: prom/alertmanager:v0.27.0
args: ["--config.file=/etc/am/alertmanager.yml", "--storage.path=/tmp/am"]
ports: [{containerPort: 9093}]
volumeMounts: [{name: cfg, mountPath: /etc/am}]
volumes: [{name: cfg, configMap: {name: am-config}}]
---
apiVersion: v1
kind: Service
metadata: {name: alertmanager, namespace: alerts}
spec: {selector: {app: alertmanager}, ports: [{port: 9093}]}We create the namespace, the ConfigMaps, the three receivers and Alertmanager:
# Alertmanager plus three receivers (oncall, slack, email): small webhook services that print what they receive
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
kubectl create namespace alerts > /dev/null
kubectl -n alerts create configmap am-config --from-file=alertmanager.yml > /dev/null && kubectl -n alerts create configmap receiver --from-file=receiver.py > /dev/null
for r in oncall slack email; do
kubectl -n alerts create deployment $r --image=python:3.12-slim -- python /app/receiver.py > /dev/null
kubectl -n alerts patch deployment $r --type=json -p '[{"op":"add","path":"/spec/template/spec/volumes","value":[{"name":"r","configMap":{"name":"receiver"}}]},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[{"name":"r","mountPath":"/app"}]}]' > /dev/null
kubectl -n alerts expose deployment $r --port=8080 > /dev/null
done
kubectl apply -f stack.yaml > /dev/null && for d in alertmanager oncall slack email; do kubectl -n alerts rollout status deploy/$d --timeout=120s | tail -1; donedeployment "alertmanager" successfully rolled out
deployment "oncall" successfully rolled out
deployment "slack" successfully rolled out
deployment "email" successfully rolled outStep 2: check the config and test routes before any alert
Why: amtool check-config catches syntax mistakes, and amtool config routes test tells you which receiver a set of labels would reach, without sending anything. Use both in your CI pipeline.
# Check the config and test the routing tree BEFORE sending anything (amtool ships inside the Alertmanager image)
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
kubectl -n alerts exec deploy/alertmanager -- amtool check-config /etc/am/alertmanager.yml | head -6
for l in "severity=critical" "team=db severity=warning" "team=web severity=warning"; do echo "$l -> $(kubectl -n alerts exec deploy/alertmanager -- amtool config routes test --config.file=/etc/am/alertmanager.yml $l)"; doneChecking '/etc/am/alertmanager.yml' SUCCESS
Found:
- global config
- route
- 1 inhibit rules
- 3 receivers
severity=critical -> oncall-pager
team=db severity=warning -> db-slack
team=web severity=warning -> team-email
Step 3: routing, three alerts, three receivers
# Send three alerts. Each must reach a different receiver
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
$AM alert add alertname=ApiDown severity=critical cluster=prod instance=api-1
$AM alert add alertname=SlowQueries severity=warning team=db cluster=prod instance=db-1
$AM alert add alertname=HighMemory severity=warning team=web cluster=prod instance=web-1
sleep 20; for r in oncall slack email; do echo "[$r] $(kubectl -n alerts logs deploy/$r | tail -1)"; done[oncall] notification: status=firing alerts=1 group={'alertname': 'ApiDown', 'cluster': 'prod'} -> ApiDown/api-1
[slack] notification: status=firing alerts=1 group={'alertname': 'SlowQueries', 'cluster': 'prod'} -> SlowQueries/db-1
[email] notification: status=firing alerts=1 group={'alertname': 'HighMemory', 'cluster': 'prod'} -> HighMemory/web-1Each alert reached exactly the receiver the tree promised.
Grouping: five alerts, one page
Five nodes go NotReady within seconds. Without grouping, the on-call engineer gets five pages for one problem:
# Grouping: five alerts of the same kind arrive within seconds. The on-call engineer gets ONE notification, not five
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
for i in 1 2 3 4 5; do $AM alert add alertname=NodeNotReady severity=critical cluster=prod instance=node-$i; done
sleep 25; kubectl -n alerts logs deploy/oncall | grep NodeNotReady | tail -1
echo "NodeNotReady notifications: $(kubectl -n alerts logs deploy/oncall | grep -c NodeNotReady)"notification: status=firing alerts=5 group={'alertname': 'NodeNotReady', 'cluster': 'prod'} -> NodeNotReady/node-1, NodeNotReady/node-2, NodeNotReady/node-3, NodeNotReady/node-4, NodeNotReady/node-5
NodeNotReady notifications: 1One notification with all five alerts inside (alerts=5), because they share alertname and cluster.
Silences: planned maintenance
Why: during maintenance, alerts are expected. A silence matches labels for a time window:
# Silence: planned maintenance on db-1 for 30 minutes. The alert still exists, but nobody is notified
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
$AM silence add instance=db-1 --duration=30m --comment="planned maintenance" --author=lab > /dev/null && $AM silence query | cut -c1-110
B=$(kubectl -n alerts logs deploy/slack | grep -c DiskFull); $AM alert add alertname=DiskFull severity=warning team=db cluster=prod instance=db-1; sleep 25
echo "DiskFull notifications to slack: $(( $(kubectl -n alerts logs deploy/slack | grep -c DiskFull) - B ))"
$AM alert query alertname=DiskFull --silenced | tail -1ID Matchers Ends At Created By Comment
3409d9e1-b2b6-49b7-a4ba-8c5c8dfa5867 instance="db-1" 2026-11-19 07:56:02 UTC lab planned maintenanc
DiskFull notifications to slack: 0
DiskFull 2026-11-19 07:26:03 UTC suppressedThe DiskFull alert exists (Alertmanager shows it as suppressed), but Slack got 0 notifications.
Inhibition: show the cause, hide the symptoms
The whole staging cluster is down, and a latency warning from the same cluster arrives:
# Inhibition: the whole staging cluster is down. Its warnings are hidden, so on-call sees the cause, not 20 symptoms
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
$AM alert add alertname=ClusterDown severity=critical cluster=staging instance=staging
$AM alert add alertname=HighLatency severity=warning team=web cluster=staging instance=web-9
sleep 25; echo "[oncall] $(kubectl -n alerts logs deploy/oncall | grep ClusterDown | tail -1)"
echo "HighLatency notifications to email: $(kubectl -n alerts logs deploy/email | grep -c HighLatency)"
$AM alert query alertname=HighLatency --inhibited | tail -1[oncall] notification: status=firing alerts=1 group={'alertname': 'ClusterDown', 'cluster': 'staging'} -> ClusterDown/staging
HighLatency notifications to email: 0
HighLatency 2026-11-19 07:26:40 UTC suppressedThe pager got ClusterDown; the email receiver got 0 HighLatency notifications, because the inhibit rule hides warnings with the same cluster label.

Break it: a capital letter
A team's exporter sends severity="Critical" with a capital C. Matchers are case sensitive, so the alert misses the pager route and falls through to the default:
# Break it: an alert labelled severity="Critical" (capital C). Matchers are case sensitive: it misses the pager route
AM="kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093"
echo "severity=Critical -> $(kubectl -n alerts exec deploy/alertmanager -- amtool config routes test --config.file=/etc/am/alertmanager.yml severity=Critical)"
$AM alert add alertname=PaymentsDown severity=Critical cluster=prod instance=pay-1; sleep 20
echo "[oncall] PaymentsDown: $(kubectl -n alerts logs deploy/oncall | grep -c PaymentsDown) [email] PaymentsDown: $(kubectl -n alerts logs deploy/email | grep -c PaymentsDown)"severity=Critical -> team-email
[oncall] PaymentsDown: 0 [email] PaymentsDown: 1A payments outage went to an email list instead of the on-call pager. Nothing failed; the routing just did not match.
Fix it: a case-insensitive matcher and a hot reload
We change the matcher to a regular expression, severity=~"(?i)critical" ((?i) means "ignore case"). Two things we learned on the way:
- Quote the whole matcher. Our first try wrote
[severity=~"(?i)critical"]without outer quotes, and that is not valid YAML inside[ ](Alertmanager refused to load it). Writing['severity=~"(?i)critical"']is valid, so the step checks the new file with a YAML parser before using it. - Reload, do not restart. A changed ConfigMap reaches the pod after up to a minute or so (each
command terminated with exit code 1line below is one check that did not see the new file yet). ThenPOST /-/reloadtells Alertmanager to read it, with no restart and no lost alerts.
# Fix it: match severity without caring about case (a regex matcher). Update the ConfigMap, wait until the pod sees
# the new file (up to a minute), then tell Alertmanager to reload it: no restart, no lost alerts
kubectl -n alerts get configmap am-config -o jsonpath='{.data.alertmanager\.yml}' | sed 's/matchers: \[severity="critical"\]/matchers: ['"'"'severity=~"(?i)critical"'"'"']/' > fixed.yml && grep "severity=~" fixed.yml
python3 -c "import yaml; yaml.safe_load(open('fixed.yml'))" && echo "fixed.yml is valid YAML"
kubectl -n alerts create configmap am-config --from-file=alertmanager.yml=fixed.yml --dry-run=client -o yaml | kubectl replace -f - > /dev/null
for i in $(seq 1 30); do kubectl -n alerts exec deploy/alertmanager -- grep -q "severity=~" /etc/am/alertmanager.yml && break; sleep 5; done; echo "new file in the pod after about $((i*5)) s"
kubectl -n alerts exec deploy/alertmanager -- wget -qO- --post-data="" http://localhost:9093/-/reload && echo "reloaded"
echo "severity=Critical -> $(kubectl -n alerts exec deploy/alertmanager -- amtool config routes test --config.file=/etc/am/alertmanager.yml severity=Critical)"
kubectl -n alerts exec deploy/alertmanager -- amtool --alertmanager.url=http://localhost:9093 alert add alertname=CheckoutDown severity=Critical cluster=prod instance=pay-2; sleep 20
echo "[oncall] CheckoutDown: $(kubectl -n alerts logs deploy/oncall | grep -c CheckoutDown) [email] CheckoutDown: $(kubectl -n alerts logs deploy/email | grep -c CheckoutDown)" - matchers: ['severity=~"(?i)critical"']
fixed.yml is valid YAML
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
command terminated with exit code 1
new file in the pod after about 70 s
reloaded
severity=Critical -> oncall-pager
[oncall] CheckoutDown: 1 [email] CheckoutDown: 0Now severity=Critical routes to oncall-pager, and a new CheckoutDown alert with a capital C reached the pager, not email.

POC: prove it works
- The config passed
amtool check-config, and route tests matched the tree (Step 2). - Three alerts reached three different receivers (Step 3).
- Five alerts became one notification; a silence and an inhibit rule each stopped a notification (Grouping, Silences, Inhibition).
- A case typo misrouted an alert; a quoted regex matcher and a hot reload fixed it (Break it, Fix it).
From webhooks to real receivers
Replace a webhook_configs entry with the real thing. These are the shapes from the Alertmanager documentation; we did not send real email, Slack or PagerDuty messages in this lab:
receivers:
- name: team-email
email_configs: [{to: "team@example.com"}] # needs global smtp_smarthost and smtp_from
- name: db-slack
slack_configs: [{channel: "#db-alerts", api_url_file: /etc/am/secrets/slack-url}]
- name: oncall-pager
pagerduty_configs: [{routing_key_file: /etc/am/secrets/pd-key}]Keep webhook URLs and keys in a Kubernetes Secret mounted as files (*_file settings), never in the ConfigMap.
Clean up (delete everything)
# Clean up: delete the namespace (Alertmanager, receivers, config)
kubectl delete namespace alerts --wait=true 2>&1 | tail -1
rm -f fixed.ymlnamespace "alerts" deletedWhen you have finished all your Kubernetes labs, delete the cluster:
gcloud container clusters delete lab-cluster --zone=us-east1-b --quietFAQ
Where do the alerts come from in real life? From Prometheus alert rules (PrometheusRule objects with the Prometheus Operator). Here we added them with amtool alert add so you can test Alertmanager on its own.
**What do group_wait, group_interval and repeat_interval do?** Wait before the first message of a new group (to collect more alerts); wait before sending an update when the group changes; resend a still-firing group after this long.
Silence or inhibit? A silence is manual and time-limited (maintenance). An inhibit rule is automatic and always on (cause hides symptoms).
How do I avoid case problems for good? Agree on label values in your alert rules, test routes with amtool config routes test in CI, and use =~"(?i)..." where you cannot control the sender.
Is one Alertmanager enough? For production, run 3 replicas in a cluster (--cluster.peer) so notifications are not lost or duplicated. This lab runs one.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
did not find expected ',' or ']' | A matcher with special characters inside [ ] without quotes | Quote each matcher: ['severity=~"(?i)critical"'] |
| An alert reaches the default receiver | No route matched (case, typo, missing label) | amtool config routes test <labels> |
| Ten pages for one problem | group_by too narrow | Group by alertname and cluster (or service) |
| A changed ConfigMap does nothing | The pod sees the file only after a delay, and Alertmanager reads it only on reload | Wait for the file, then POST /-/reload |
| A silence does not work | The matcher does not match the alert's labels exactly | amtool silence query and compare labels |
What next
- Make alerts fire from real metrics: Prometheus and Grafana on Kubernetes from Scratch.
- See alerts around a real ML service: MLOps for Beginners: From Notebook to Production.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamObservability Interview Questions with Hands-On Proof
What you build: 15 observability interview answers, each proved with a real command on GKE with Prometheus: rate, p99, cardinality, SLOs, alerts. Every step tested.
#google
CareerByteCode TeamGoogle Cloud Managed Prometheus: No Servers to Run
What you build: Prometheus metrics from a GKE app in Google Cloud Managed Service for Prometheus with one PodMonitoring, PromQL queries and a silent typo found. Every step tested.
#google-cloud
CareerByteCode TeamLogs to BigQuery: Analyse a Month of Logs with SQL
What you build: a log sink from Cloud Logging to BigQuery and four SQL reports on a month of web traffic, errors and slow pages, in your own Google Cloud account. Every step tested.
#google-cloud
CareerByteCode TeamOpenTelemetry for Beginners: Traces, Metrics and Logs
What you build: an instrumented app sending traces to Cloud Trace, in your own Google Cloud account. Every step, command and screenshot explained for intermediates.
💬 Comments (0)
Top · NewestSign in to comment