Grafana Dashboards That Engineers Actually Use
What you build: a Grafana dashboard rebuilt around RED and USE, reviewed like code, tested against a real bad release on GKE. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
Most first dashboards look busy and still do not answer the one question an engineer has at 3 AM: is the service OK, and if not, where does it hurt? In this lab you load a typical "show everything" dashboard, review it like code with a small script, rebuild it around two questions, and then break the app to see which dashboard tells you the truth.
Who it is for: anyone who has installed Grafana once and now wants dashboards people actually use. If Prometheus and Grafana are new to you, do Prometheus and Grafana on Kubernetes from Scratch first. Time: about 1.5 hours. Cost: a small GKE cluster for about 20 minutes, covered by the free trial credit. Delete it at the end.
Every command below was run for real by the CareerByteCode Team, and every output and Grafana screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl, helm, jq and python3); Cloud Shell in your browser has the same tools.
Part A: what makes a dashboard useful
A dashboard is useful when a tired person can read it in 10 seconds. That needs four things:
- One question per row. Our rows ask "Is it working for users?" and "Is it healthy inside?". Everything else goes to another dashboard.
- The right method. For a service, use RED: Rate (requests per second), Errors (share of failed requests), Duration (how long requests take). For resources such as CPU and memory, use USE: Utilisation, Saturation, Errors.
- Units and thresholds.
0.0665means nothing;66.5 msdoes. A colour (green, orange, red) tells you what is normal without remembering numbers. - Percentiles, not averages. p95 means "95 of every 100 requests were faster than this". An average hides the slow requests your users complain about.
Other words you will use:
- Panel: one box on the dashboard (a graph or a big number, called a stat).
- Variable: a drop-down at the top (Namespace, Pod) that changes every query, so one dashboard works for many apps.
- Provisioning: Grafana loads the dashboard from a file (here a Kubernetes ConfigMap), so the dashboard lives in Git like code.
rate(): turns a counter that only goes up (total requests) into "per second". A raw counter on a graph is almost never useful.

Before you start
- A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
- Set your project in Cloud Shell and turn on the Kubernetes Engine API (replace the text in quotes with your own project id):
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com- One small GKE cluster. We created it once for this batch of Kubernetes labs (if you still have a
lab-clusterfrom another of our Kubernetes labs, use it and skip this command):
gcloud container clusters create lab-cluster \
--zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
--disk-type=pd-standard --disk-size=50 --release-channel=regular \
--no-enable-autoupgrade --enable-dataplane-v2- Download nothing else: every file of the lab is shown below. Save each one with the name given.
Part B: the lab
Step 1: the app and the monitoring stack
Why: we need an app with real traffic. podinfo is a small web app that publishes Prometheus metrics. The load pod calls it 5 times a second, and its second container (slow) calls /delay/1, a page that takes 1 second, every 2 seconds. So the app has both fast and slow requests, like a real one. Save this as app.yaml:
# podinfo: a small web app that publishes Prometheus metrics on /metrics, plus a load pod that calls it all the time
apiVersion: apps/v1
kind: Deployment
metadata: {name: podinfo, labels: {app: podinfo}}
spec:
replicas: 2
selector: {matchLabels: {app: podinfo}}
template:
metadata: {labels: {app: podinfo}}
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.7.1
ports: [{name: http, containerPort: 9898}]
resources: {requests: {cpu: 20m, memory: 32Mi}}
---
apiVersion: v1
kind: Service
metadata: {name: podinfo, labels: {app: podinfo}}
spec:
selector: {app: podinfo}
ports: [{name: http, port: 9898, targetPort: 9898}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: load}
spec:
replicas: 1
selector: {matchLabels: {app: load}}
template:
metadata: {labels: {app: load}}
spec:
containers:
- name: load
image: curlimages/curl:8.10.1
env: [{name: ERROR_PATH, value: ""}]
command: ["sh", "-c", "while true; do curl -s -o /dev/null http://podinfo:9898/; [ -n \"$ERROR_PATH\" ] && curl -s -o /dev/null http://podinfo:9898$ERROR_PATH; sleep 0.2; done"]
resources: {requests: {cpu: 10m, memory: 16Mi}}
- name: slow
image: curlimages/curl:8.10.1
command: ["sh", "-c", "while true; do curl -s -o /dev/null http://podinfo:9898/delay/1; sleep 2; done"]
resources: {requests: {cpu: 10m, memory: 16Mi}}The stack is kube-prometheus-stack (Prometheus, Grafana, kube-state-metrics, node-exporter). Our settings file switches off what we do not need and opens Grafana read-only for viewers without a password, which is fine only inside a lab that is reachable through port-forward. Save it as values.yaml:
# kube-prometheus-stack settings for this lab: small, short retention, Grafana open read-only for viewers (lab only)
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
retention: 6h
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeEtcd: {enabled: false}
kubeProxy: {enabled: false}
alertmanager: {enabled: false}
grafana:
defaultDashboardsEnabled: false
grafana.ini:
auth.anonymous: {enabled: true, org_role: Viewer}
users: {default_theme: light}And a ServiceMonitor tells Prometheus to scrape podinfo every 15 seconds (servicemonitor.yaml):
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: {name: podinfo, namespace: shop}
spec:
selector: {matchLabels: {app: podinfo}}
endpoints: [{port: http, interval: 15s}]# The sample app and the monitoring stack (Prometheus + Grafana) with Helm
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
kubectl create namespace shop > /dev/null && kubectl apply -n shop -f app.yaml > /dev/null && kubectl -n shop rollout status deploy/podinfo deploy/load --timeout=180s | tail -1
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts > /dev/null; helm repo update > /dev/null
helm install monitoring prometheus-community/kube-prometheus-stack --version 66.3.1 -n monitoring --create-namespace -f values.yaml --wait --timeout 10m | grep -E "^(NAME|STATUS)"
kubectl apply -f servicemonitor.yaml
nohup kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 > pf-prom.log 2>&1 &
nohup kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 > pf-grafana.log 2>&1 &
sleep 60; curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.labels.namespace=="shop") | "target " + .labels.pod + ": " + .health'deployment "load" successfully rolled out
NAME: monitoring
NAMESPACE: monitoring
STATUS: deployed
servicemonitor.monitoring.coreos.com/podinfo createdIn our run the last line printed nothing: 60 seconds was not yet enough for Prometheus to pick up the new ServiceMonitor. The review in Step 3 shows the podinfo queries returning data, so the scrape started soon after. If your list is empty too, wait a minute and run the curl ... /api/v1/targets line again: you should see each podinfo pod with up.
Step 2: the "show everything" dashboard
Why: to see what goes wrong, start where most teams start. This dashboard has five panels: raw request counters, memory in bytes, a "Latency" number, an "Errors" number and CPU for everything. Save the whole file as bad.json (Grafana's JSON model, one panel per line):
{
"title": "Podinfo - everything",
"uid": "podinfo-bad",
"time": {"from": "now-15m", "to": "now"},
"refresh": "10s",
"panels": [
{"type": "timeseries", "title": "Requests", "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "http_request_duration_seconds_count{namespace=\"shop\"}", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
{"type": "timeseries", "title": "Memory", "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "process_resident_memory_bytes{namespace=\"shop\"}", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
{"type": "stat", "title": "Latency", "gridPos": {"x": 0, "y": 8, "w": 6, "h": 6}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(http_request_duration_seconds_sum{namespace=\"shop\"}) / sum(http_request_duration_seconds_count{namespace=\"shop\"})", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
{"type": "stat", "title": "Errors", "gridPos": {"x": 6, "y": 8, "w": 6, "h": 6}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(http_requests_total{namespace=\"shop\",code=\"500\"})", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
{"type": "timeseries", "title": "CPU", "gridPos": {"x": 12, "y": 8, "w": 12, "h": 6}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "rate(process_cpu_seconds_total[5m])", "legendFormat": ""}], "fieldConfig": {"defaults": {}}}
]
}We load it the provisioning way: a ConfigMap with the label grafana_dashboard=1. Grafana's helper (the sidecar) picks up every ConfigMap with that label.
# Load the first dashboard: five panels that "show everything"
kubectl -n monitoring create configmap dash-bad --from-file=bad.json > /dev/null && kubectl -n monitoring label configmap dash-bad grafana_dashboard=1 > /dev/null
sleep 30; curl -s "http://localhost:3000/api/search?query=Podinfo" | jq -r '.[] | "loaded: " + .title + " (uid " + .uid + ")"'loaded: Podinfo - everything (uid podinfo-bad)
Look at it as the person on call: "Requests" only climbs (it is a total, not a rate), the legend is a wall of labels, "Latency" is 0.0665 of what?, "Errors" says No data, and "CPU" mixes Grafana, the config reloader and everything else.
Step 3: review the dashboard like code
Why: a dashboard is code, so review it like code. This small Python script, lint.py, checks every panel for a unit, a description, and whether its query returns data from Prometheus right now:
# lint.py <dashboard.json>: review a dashboard like code. For every panel: does it have a unit, a description,
# and does each query return data from Prometheus right now? Exit code 1 when any panel has a problem.
import json, sys, urllib.parse, urllib.request
d = json.load(open(sys.argv[1]))
subs = {'$namespace': 'shop', '$pod': '.*', '$__rate_interval': '1m'}
bad = 0
print(f'dashboard: {d["title"]}')
for p in d['panels']:
if p['type'] == 'row':
continue
problems = []
if not p.get('fieldConfig', {}).get('defaults', {}).get('unit'):
problems.append('no unit')
if not p.get('description'):
problems.append('no description')
for t in p['targets']:
q = t['expr']
for k, v in subs.items():
q = q.replace(k, v)
r = json.load(urllib.request.urlopen('http://localhost:9090/api/v1/query?' + urllib.parse.urlencode({'query': q})))
n = len(r['data']['result'])
if n == 0:
problems.append('query returns no data')
elif n > 10:
problems.append(f'{n} lines on one panel')
bad += bool(problems)
print(f' {"FAIL" if problems else "ok "} {p["title"]:<36} {", ".join(problems)}')
print(f'{bad} of {sum(p["type"] != "row" for p in d["panels"])} panels need work')
sys.exit(1 if bad else 0)# Review it like code: unit? description? does every query return data?
python3 lint.py bad.jsondashboard: Podinfo - everything
FAIL Requests no unit, no description
FAIL Memory no unit, no description
FAIL Latency no unit, no description
FAIL Errors no unit, no description, query returns no data
FAIL CPU no unit, no description, 13 lines on one panel
5 of 5 panels need work
All 5 panels fail. The "Errors" panel is the dangerous one: the metric http_requests_total does not exist in podinfo (it calls its metric http_request_duration_seconds_count with a status label), so the panel shows No data, and "no data" looks like "no errors".
Step 4: rebuild it around two questions
Why: the new dashboard (good.json) has two rows:
- Is it working for users? Requests per second, Errors (5xx share in percent: orange from 1, red from 5), the slowest 5 percent (p95: orange from 0.5 s, red from 1 s), Pods ready, then two graphs: requests by status and duration p50, p95, p99.
- Is it healthy inside? CPU used as a share of what the pod requested, memory in use, restarts in the last hour.
Every query uses rate(...[$__rate_interval]) (Grafana picks a sensible window) and the variables $namespace and $pod. Two of its panels, as an example (shortened):
{"type": "stat", "title": "Errors (5xx share)", "description": "Share of answers that were server errors. Orange from 1 percent, red from 5 percent.",
"targets": [{"expr": "100 * (sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\", status=~\"5..\"}[$__rate_interval])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))"}],
"fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 5}]}}}}
{"type": "stat", "title": "Slowest 5 percent (p95)", "description": "95 of every 100 requests were faster than this. Orange from 0.5 s, red from 1 s.",
"targets": [{"expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))"}],
"fieldConfig": {"defaults": {"unit": "s"}}}The whole file, one panel per line. Save it as good.json:
{
"title": "Podinfo - service health",
"uid": "podinfo-good",
"time": {"from": "now-15m", "to": "now"},
"refresh": "10s",
"templating": {"list": [
{"name": "namespace", "label": "Namespace", "type": "query", "datasource": {"type": "prometheus", "uid": "prometheus"}, "query": {"query": "label_values(http_request_duration_seconds_count, namespace)", "refId": "ns"}, "refresh": 1, "current": {"text": "shop", "value": "shop"}},
{"name": "pod", "label": "Pod", "type": "query", "datasource": {"type": "prometheus", "uid": "prometheus"}, "query": {"query": "label_values(http_request_duration_seconds_count{namespace=\"$namespace\"}, pod)", "refId": "pod"}, "refresh": 2, "includeAll": true, "multi": true, "allValue": ".*", "current": {"text": "All", "value": "$__all"}}
]},
"panels": [
{"type": "row", "title": "Is it working for users? (requests, errors, duration)", "gridPos": {"x": 0, "y": 0, "w": 24, "h": 1}, "collapsed": false, "panels": []},
{"type": "stat", "title": "Requests per second", "gridPos": {"x": 0, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "reqps"}}, "description": "All requests the app answered, per second."},
{"type": "stat", "title": "Errors (5xx share)", "gridPos": {"x": 6, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "100 * (sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\", status=~\"5..\"}[$__rate_interval])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 5}]}}}, "description": "Share of answers that were server errors. Orange from 1 percent, red from 5 percent.", "options": {"colorMode": "background"}},
{"type": "stat", "title": "Slowest 5 percent (p95)", "gridPos": {"x": 12, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "s", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.5}, {"color": "red", "value": 1}]}}}, "description": "95 of every 100 requests were faster than this. Orange from 0.5 s, red from 1 s.", "options": {"colorMode": "background"}},
{"type": "stat", "title": "Pods ready", "gridPos": {"x": 18, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(kube_pod_status_ready{namespace=\"$namespace\", pod=~\"podinfo.*\", condition=\"true\"})", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "none"}}, "description": "Podinfo pods that pass their readiness check."},
{"type": "timeseries", "title": "Requests per second by status", "gridPos": {"x": 0, "y": 6, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum by (status) (rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))", "legendFormat": "{{status}}"}], "fieldConfig": {"defaults": {"unit": "reqps"}}, "description": "One line per HTTP status code."},
{"type": "timeseries", "title": "Duration: p50, p95, p99", "gridPos": {"x": 12, "y": 6, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "histogram_quantile(0.5, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": "p50"}, {"refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": "p95"}, {"refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": "p99"}], "fieldConfig": {"defaults": {"unit": "s"}}, "description": "The typical request (p50) and the slow ones (p95, p99)."},
{"type": "row", "title": "Is it healthy inside? (use and saturation)", "gridPos": {"x": 0, "y": 14, "w": 24, "h": 1}, "collapsed": false, "panels": []},
{"type": "timeseries", "title": "CPU used, share of the request", "gridPos": {"x": 0, "y": 15, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "100 * sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\"}[$__rate_interval])) / sum by (pod) (kube_pod_container_resource_requests{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\", resource=\"cpu\"})", "legendFormat": "{{pod}}"}], "fieldConfig": {"defaults": {"unit": "percent"}}, "description": "Above 100 percent the pod uses more CPU than it asked for."},
{"type": "timeseries", "title": "Memory in use", "gridPos": {"x": 12, "y": 15, "w": 8, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum by (pod) (container_memory_working_set_bytes{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\"})", "legendFormat": "{{pod}}"}], "fieldConfig": {"defaults": {"unit": "bytes"}}, "description": "Working set memory: what the kernel counts against the limit."},
{"type": "stat", "title": "Restarts (last hour)", "gridPos": {"x": 20, "y": 15, "w": 4, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(increase(kube_pod_container_status_restarts_total{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\"}[1h]))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "none", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}}, "description": "Container restarts. Anything above 0 needs a look.", "options": {"colorMode": "background"}}
]
}or vector(0) makes the Errors panel show 0 when there are no errors, instead of No data. We run the same review first, then load it:
# The rebuilt dashboard: two questions (working for users? healthy inside?), units, thresholds, a Pod picker
python3 lint.py good.json
kubectl -n monitoring create configmap dash-good --from-file=good.json > /dev/null && kubectl -n monitoring label configmap dash-good grafana_dashboard=1 > /dev/null
sleep 30; curl -s "http://localhost:3000/api/dashboards/uid/podinfo-good" | jq -r '"loaded: " + .dashboard.title + ", version " + (.dashboard.version | tostring) + ", " + (.dashboard.panels | length | tostring) + " panels, variables: " + ([.dashboard.templating.list[].name] | join(", "))'dashboard: Podinfo - service health
ok Requests per second
ok Errors (5xx share)
ok Slowest 5 percent (p95)
ok Pods ready
ok Requests per second by status
ok Duration: p50, p95, p99
ok CPU used, share of the request
ok Memory in use
ok Restarts (last hour)
0 of 9 panels need work
loaded: Podinfo - service health, version 1, 11 panels, variables: namespace, pod
Read it in 10 seconds: traffic is fine, no errors, but p95 is red at 1.38 s. The duration graph shows why: p50 (the typical request) is a few milliseconds, while p95 and p99 are above 1 second. That is our slow container calling /delay/1. The old dashboard's "Latency 0.0665" (an average) completely hid it.
Break it: a bad release
We change the load pod so it also calls /status/500, a page that always fails, as if a bad release went out. Then we read the Errors panel of both dashboards after 90 seconds:
# Break it: a bad release. The load pod now also calls /status/500, a page that always fails
kubectl -n shop set env deploy/load -c load ERROR_PATH=/status/500 > /dev/null && kubectl -n shop rollout status deploy/load --timeout=90s | tail -1
sleep 90
Q='100 * sum(rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m])) / sum(rate(http_request_duration_seconds_count{namespace="shop"}[1m]))'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '"good dashboard, Errors panel: " + (.data.result[0].value[1] | tonumber | round | tostring) + " percent (red from 5)"'
Q='sum(http_requests_total{namespace="shop",code="500"})'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '"bad dashboard, Errors panel: " + (if (.data.result | length) == 0 then "No data" else .data.result[0].value[1] end)'📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →deployment "load" successfully rolled out
good dashboard, Errors panel: 47 percent (red from 5)
bad dashboard, Errors panel: No data

The rebuilt dashboard turns red at once and shows a new status 500 line. The first dashboard still says No data: on that dashboard the incident is invisible.
One more lesson in the screenshot: p95 got better (4.97 ms, green) during the incident. The failing calls answer instantly, so there are many more fast requests and the slow ones drop out of the top 5 percent. Never judge a release by latency alone; read Errors and Duration together. That is why RED puts them side by side.
Fix it: roll back
We roll the load pod back to its old settings and watch the same number the Errors panel shows:
# Fix it: roll the bad release back, then watch the Errors panel go back to 0
Q='100 * (sum(rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace="shop"}[1m]))'
kubectl -n shop set env deploy/load -c load ERROR_PATH- > /dev/null && kubectl -n shop rollout status deploy/load --timeout=90s | tail -1
for i in $(seq 1 12); do sleep 15; V=$(curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '.data.result[0].value[1] | tonumber | round'); echo "after $((i*15))s: errors $V percent"; [ "$V" = 0 ] && break; donedeployment "load" successfully rolled out
after 15s: errors 42 percent
after 30s: errors 37 percent
after 45s: errors 30 percent
after 60s: errors 22 percent
after 75s: errors 9 percent
after 90s: errors 0 percent
The share falls step by step because the panel looks at the last minute: after 90 seconds every request in the window is good again.

Dashboards as code
Why: the JSON file in Git is the truth. Because Grafana loaded the dashboard from a file, it marks it provisioned: a change made by hand in the Grafana screen cannot be saved over it, and a reviewer can see every change in a pull request.
# Dashboards as code: the file in Git is the truth. A change made by hand in the Grafana UI is lost on the next load
curl -s "http://localhost:3000/api/dashboards/uid/podinfo-good" | jq '.dashboard | del(.id, .version, .iteration)' > live.json
python3 - <<'P'
import json
a, b = json.load(open('good.json')), json.load(open('live.json'))
same = [p['title'] for p in a['panels']] == [p['title'] for p in b['panels']]
print('panels in Grafana match the file in Git:', same)
P
curl -s "http://localhost:3000/api/dashboards/uid/podinfo-good" | jq -r '"provisioned from a file (read-only in the UI): " + (.meta.provisioned | tostring)'panels in Grafana match the file in Git: True
provisioned from a file (read-only in the UI): truePOC: prove it works
- The review script failed the first dashboard on all 5 panels and passed the rebuilt one on all 9 (Steps 3 and 4).
- During the bad release, the rebuilt dashboard showed 47 percent errors in red while the first one showed No data (Break it).
- After the rollback the errors went back to 0 percent within 90 seconds (Fix it).
- Grafana reports the dashboard as provisioned from the file, matching Git (Dashboards as code).
Clean up (delete everything)
Why: the stack is free only while the cluster runs, and the cluster is the cost. Helm leaves the Prometheus Operator's custom types behind on purpose, so we remove them too.
# Clean up: the stack, the app and the Prometheus Operator types
pkill -f "[p]ort-forward svc/monitoring"
helm uninstall monitoring -n monitoring --wait 2>&1 | tail -1
kubectl delete namespace shop monitoring --wait=true 2>&1 | tail -2
kubectl get crd -o name | grep monitoring.coreos.com | xargs -r kubectl delete > /dev/null && echo "Prometheus Operator types removed"
echo "namespaces left from this lab: $(kubectl get ns shop monitoring --no-headers 2>/dev/null | wc -l)"
rm -f ~/app.yaml ~/values.yaml ~/servicemonitor.yaml ~/bad.json ~/good.json ~/lint.py ~/live.json ~/pf-prom.log ~/pf-grafana.log # prints nothingrelease "monitoring" uninstalled
namespace "shop" deleted
namespace "monitoring" deleted
Prometheus Operator types removed
namespaces left from this lab: 0
When you have finished all your Kubernetes labs, delete the cluster:
gcloud container clusters delete lab-cluster --zone=us-east1-b --quietFAQ
How many panels should a dashboard have? As few as answer its question. Our service dashboard has 9. Put deep-dive panels (per-path latency, garbage collection, queue sizes) on a second dashboard and link to it.
RED or USE? Both, for different things. RED tells you whether users are hurting; USE tells you whether a resource (CPU, memory, disk) is the reason.
Why $__rate_interval and not [1m]? Grafana sets it from the scrape interval and the time range, so the graph stays smooth when you zoom out to 7 days. In our review script we replaced it with 1m.
Why not an average latency? An average mixes many fast requests with a few slow ones and shows a number nobody experienced. Our average was 0.0665 s while 1 in 20 requests took more than 1.3 s.
Can I still edit the dashboard in the screen? Yes, to try things. Then copy the JSON (Share > Export) into Git and let provisioning load it.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
| A panel shows No data | Wrong metric name or label, or no traffic | Run the query in Prometheus first; add or vector(0) where 0 is a real answer |
| A graph only climbs | A counter drawn without rate() | Wrap it: sum(rate(metric[$__rate_interval])) |
| A legend as long as the screen | Every label printed | Aggregate (sum by (status)) and set legendFormat |
| The dashboard does not appear | The ConfigMap has no grafana_dashboard=1 label or is in another namespace | Label it and keep it in the namespace the sidecar watches |
| Grafana says the dashboard cannot be saved | It is provisioned from a file | Change the file in Git and apply it again |
| The Pod drop-down is empty | The variable query has no data for that namespace | Check the Namespace drop-down first |
What next
- Learn the stack itself: Prometheus and Grafana on Kubernetes from Scratch.
- Put a service mesh under your metrics: Istio Service Mesh for Beginners.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamPrometheus and Grafana on Kubernetes from Scratch
What you build: Prometheus, Grafana and Alertmanager on GKE with your first dashboard and an alert that really fires and clears, in your own Google Cloud account. Every step tested.
#beginners
CareerByteCode TeamMonitor Linux Servers with Node Exporter and Ops Agent
What you build: CPU, memory and disk alerts for VMs, in your own Google Cloud account. Every step, command and screenshot explained for beginners.
#beginners
CareerByteCode TeamCloud Logging Hands-On: Find Any Error in Seconds
What you build: log queries, log-based metrics and sinks, in your own Google Cloud account. Every step, command and screenshot explained for beginners.
#alertmanager
CareerByteCode TeamAlertmanager: Route Alerts to Email, Slack and On-Call
What you build: Alertmanager on GKE with routing, grouping, silences and inhibition tested, a case typo that misroutes a page, and a hot-reload fix. Every step tested.
💬 Comments (0)
Top · NewestSign in to comment