CareerByteCode
ByteBlogby CareerByteCode

Grafana Dashboards That Engineers Actually Use

What you build: a Grafana dashboard rebuilt around RED and USE, reviewed like code, tested against a real bad release on GKE. Every step tested.

Grafana Dashboards That Engineers Actually Use

✍️ Translate this blog into Hindi, Tamil or Telugu

Most first dashboards look busy and still do not answer the one question an engineer has at 3 AM: is the service OK, and if not, where does it hurt? In this lab you load a typical "show everything" dashboard, review it like code with a small script, rebuild it around two questions, and then break the app to see which dashboard tells you the truth.

Who it is for: anyone who has installed Grafana once and now wants dashboards people actually use. If Prometheus and Grafana are new to you, do Prometheus and Grafana on Kubernetes from Scratch first. Time: about 1.5 hours. Cost: a small GKE cluster for about 20 minutes, covered by the free trial credit. Delete it at the end.

Every command below was run for real by the CareerByteCode Team, and every output and Grafana screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl, helm, jq and python3); Cloud Shell in your browser has the same tools.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: what makes a dashboard useful

A dashboard is useful when a tired person can read it in 10 seconds. That needs four things:

  • One question per row. Our rows ask "Is it working for users?" and "Is it healthy inside?". Everything else goes to another dashboard.
  • The right method. For a service, use RED: Rate (requests per second), Errors (share of failed requests), Duration (how long requests take). For resources such as CPU and memory, use USE: Utilisation, Saturation, Errors.
  • Units and thresholds. 0.0665 means nothing; 66.5 ms does. A colour (green, orange, red) tells you what is normal without remembering numbers.
  • Percentiles, not averages. p95 means "95 of every 100 requests were faster than this". An average hides the slow requests your users complain about.

Other words you will use:

  • Panel: one box on the dashboard (a graph or a big number, called a stat).
  • Variable: a drop-down at the top (Namespace, Pod) that changes every query, so one dashboard works for many apps.
  • Provisioning: Grafana loads the dashboard from a file (here a Kubernetes ConfigMap), so the dashboard lives in Git like code.
  • rate(): turns a counter that only goes up (total requests) into "per second". A raw counter on a graph is almost never useful.
How the lab fits together
The app publishes metrics, Prometheus stores them, Grafana draws two dashboards from JSON files

Before you start

  1. A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
  2. Set your project in Cloud Shell and turn on the Kubernetes Engine API (replace the text in quotes with your own project id):
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com
  1. One small GKE cluster. We created it once for this batch of Kubernetes labs (if you still have a lab-cluster from another of our Kubernetes labs, use it and skip this command):
gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --enable-dataplane-v2
  1. Download nothing else: every file of the lab is shown below. Save each one with the name given.

Part B: the lab

Step 1: the app and the monitoring stack

Why: we need an app with real traffic. podinfo is a small web app that publishes Prometheus metrics. The load pod calls it 5 times a second, and its second container (slow) calls /delay/1, a page that takes 1 second, every 2 seconds. So the app has both fast and slow requests, like a real one. Save this as app.yaml:

# podinfo: a small web app that publishes Prometheus metrics on /metrics, plus a load pod that calls it all the time
apiVersion: apps/v1
kind: Deployment
metadata: {name: podinfo, labels: {app: podinfo}}
spec:
  replicas: 2
  selector: {matchLabels: {app: podinfo}}
  template:
    metadata: {labels: {app: podinfo}}
    spec:
      containers:
      - name: podinfo
        image: ghcr.io/stefanprodan/podinfo:6.7.1
        ports: [{name: http, containerPort: 9898}]
        resources: {requests: {cpu: 20m, memory: 32Mi}}
---
apiVersion: v1
kind: Service
metadata: {name: podinfo, labels: {app: podinfo}}
spec:
  selector: {app: podinfo}
  ports: [{name: http, port: 9898, targetPort: 9898}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: load}
spec:
  replicas: 1
  selector: {matchLabels: {app: load}}
  template:
    metadata: {labels: {app: load}}
    spec:
      containers:
      - name: load
        image: curlimages/curl:8.10.1
        env: [{name: ERROR_PATH, value: ""}]
        command: ["sh", "-c", "while true; do curl -s -o /dev/null http://podinfo:9898/; [ -n \"$ERROR_PATH\" ] && curl -s -o /dev/null http://podinfo:9898$ERROR_PATH; sleep 0.2; done"]
        resources: {requests: {cpu: 10m, memory: 16Mi}}
      - name: slow
        image: curlimages/curl:8.10.1
        command: ["sh", "-c", "while true; do curl -s -o /dev/null http://podinfo:9898/delay/1; sleep 2; done"]
        resources: {requests: {cpu: 10m, memory: 16Mi}}

The stack is kube-prometheus-stack (Prometheus, Grafana, kube-state-metrics, node-exporter). Our settings file switches off what we do not need and opens Grafana read-only for viewers without a password, which is fine only inside a lab that is reachable through port-forward. Save it as values.yaml:

# kube-prometheus-stack settings for this lab: small, short retention, Grafana open read-only for viewers (lab only)
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    retention: 6h
kubeControllerManager: {enabled: false}
kubeScheduler: {enabled: false}
kubeEtcd: {enabled: false}
kubeProxy: {enabled: false}
alertmanager: {enabled: false}
grafana:
  defaultDashboardsEnabled: false
  grafana.ini:
    auth.anonymous: {enabled: true, org_role: Viewer}
    users: {default_theme: light}

And a ServiceMonitor tells Prometheus to scrape podinfo every 15 seconds (servicemonitor.yaml):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: {name: podinfo, namespace: shop}
spec:
  selector: {matchLabels: {app: podinfo}}
  endpoints: [{port: http, interval: 15s}]
# The sample app and the monitoring stack (Prometheus + Grafana) with Helm
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
kubectl create namespace shop > /dev/null && kubectl apply -n shop -f app.yaml > /dev/null && kubectl -n shop rollout status deploy/podinfo deploy/load --timeout=180s | tail -1
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts > /dev/null; helm repo update > /dev/null
helm install monitoring prometheus-community/kube-prometheus-stack --version 66.3.1 -n monitoring --create-namespace -f values.yaml --wait --timeout 10m | grep -E "^(NAME|STATUS)"
kubectl apply -f servicemonitor.yaml
nohup kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 > pf-prom.log 2>&1 &
nohup kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 > pf-grafana.log 2>&1 &
sleep 60; curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.labels.namespace=="shop") | "target " + .labels.pod + ": " + .health'
deployment "load" successfully rolled out
NAME: monitoring
NAMESPACE: monitoring
STATUS: deployed
servicemonitor.monitoring.coreos.com/podinfo created

In our run the last line printed nothing: 60 seconds was not yet enough for Prometheus to pick up the new ServiceMonitor. The review in Step 3 shows the podinfo queries returning data, so the scrape started soon after. If your list is empty too, wait a minute and run the curl ... /api/v1/targets line again: you should see each podinfo pod with up.

Step 2: the "show everything" dashboard

Why: to see what goes wrong, start where most teams start. This dashboard has five panels: raw request counters, memory in bytes, a "Latency" number, an "Errors" number and CPU for everything. Save the whole file as bad.json (Grafana's JSON model, one panel per line):

{
 "title": "Podinfo - everything",
 "uid": "podinfo-bad",
 "time": {"from": "now-15m", "to": "now"},
 "refresh": "10s",
 "panels": [
  {"type": "timeseries", "title": "Requests", "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "http_request_duration_seconds_count{namespace=\"shop\"}", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
  {"type": "timeseries", "title": "Memory", "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "process_resident_memory_bytes{namespace=\"shop\"}", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
  {"type": "stat", "title": "Latency", "gridPos": {"x": 0, "y": 8, "w": 6, "h": 6}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(http_request_duration_seconds_sum{namespace=\"shop\"}) / sum(http_request_duration_seconds_count{namespace=\"shop\"})", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
  {"type": "stat", "title": "Errors", "gridPos": {"x": 6, "y": 8, "w": 6, "h": 6}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(http_requests_total{namespace=\"shop\",code=\"500\"})", "legendFormat": ""}], "fieldConfig": {"defaults": {}}},
  {"type": "timeseries", "title": "CPU", "gridPos": {"x": 12, "y": 8, "w": 12, "h": 6}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "rate(process_cpu_seconds_total[5m])", "legendFormat": ""}], "fieldConfig": {"defaults": {}}}
 ]
}

We load it the provisioning way: a ConfigMap with the label grafana_dashboard=1. Grafana's helper (the sidecar) picks up every ConfigMap with that label.

# Load the first dashboard: five panels that "show everything"
kubectl -n monitoring create configmap dash-bad --from-file=bad.json > /dev/null && kubectl -n monitoring label configmap dash-bad grafana_dashboard=1 > /dev/null
sleep 30; curl -s "http://localhost:3000/api/search?query=Podinfo" | jq -r '.[] | "loaded: " + .title + " (uid " + .uid + ")"'
loaded: Podinfo - everything (uid podinfo-bad)
The first dashboard
Real Grafana screen: counters that only climb, memory as 42000000, Latency 0.0665 with no unit, Errors: No data, CPU of every pod in the cluster

Look at it as the person on call: "Requests" only climbs (it is a total, not a rate), the legend is a wall of labels, "Latency" is 0.0665 of what?, "Errors" says No data, and "CPU" mixes Grafana, the config reloader and everything else.

Step 3: review the dashboard like code

Why: a dashboard is code, so review it like code. This small Python script, lint.py, checks every panel for a unit, a description, and whether its query returns data from Prometheus right now:

# lint.py <dashboard.json>: review a dashboard like code. For every panel: does it have a unit, a description,
# and does each query return data from Prometheus right now? Exit code 1 when any panel has a problem.
import json, sys, urllib.parse, urllib.request
d = json.load(open(sys.argv[1]))
subs = {'$namespace': 'shop', '$pod': '.*', '$__rate_interval': '1m'}
bad = 0
print(f'dashboard: {d["title"]}')
for p in d['panels']:
    if p['type'] == 'row':
        continue
    problems = []
    if not p.get('fieldConfig', {}).get('defaults', {}).get('unit'):
        problems.append('no unit')
    if not p.get('description'):
        problems.append('no description')
    for t in p['targets']:
        q = t['expr']
        for k, v in subs.items():
            q = q.replace(k, v)
        r = json.load(urllib.request.urlopen('http://localhost:9090/api/v1/query?' + urllib.parse.urlencode({'query': q})))
        n = len(r['data']['result'])
        if n == 0:
            problems.append('query returns no data')
        elif n > 10:
            problems.append(f'{n} lines on one panel')
    bad += bool(problems)
    print(f'  {"FAIL" if problems else "ok  "}  {p["title"]:<36} {", ".join(problems)}')
print(f'{bad} of {sum(p["type"] != "row" for p in d["panels"])} panels need work')
sys.exit(1 if bad else 0)
# Review it like code: unit? description? does every query return data?
python3 lint.py bad.json
dashboard: Podinfo - everything
  FAIL  Requests                             no unit, no description
  FAIL  Memory                               no unit, no description
  FAIL  Latency                              no unit, no description
  FAIL  Errors                               no unit, no description, query returns no data
  FAIL  CPU                                  no unit, no description, 13 lines on one panel
5 of 5 panels need work
The review script
Step 3: the review script fails all 5 panels of the first dashboard

All 5 panels fail. The "Errors" panel is the dangerous one: the metric http_requests_total does not exist in podinfo (it calls its metric http_request_duration_seconds_count with a status label), so the panel shows No data, and "no data" looks like "no errors".

Step 4: rebuild it around two questions

Why: the new dashboard (good.json) has two rows:

  • Is it working for users? Requests per second, Errors (5xx share in percent: orange from 1, red from 5), the slowest 5 percent (p95: orange from 0.5 s, red from 1 s), Pods ready, then two graphs: requests by status and duration p50, p95, p99.
  • Is it healthy inside? CPU used as a share of what the pod requested, memory in use, restarts in the last hour.

Every query uses rate(...[$__rate_interval]) (Grafana picks a sensible window) and the variables $namespace and $pod. Two of its panels, as an example (shortened):

{"type": "stat", "title": "Errors (5xx share)", "description": "Share of answers that were server errors. Orange from 1 percent, red from 5 percent.",
 "targets": [{"expr": "100 * (sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\", status=~\"5..\"}[$__rate_interval])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))"}],
 "fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 5}]}}}}
{"type": "stat", "title": "Slowest 5 percent (p95)", "description": "95 of every 100 requests were faster than this. Orange from 0.5 s, red from 1 s.",
 "targets": [{"expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))"}],
 "fieldConfig": {"defaults": {"unit": "s"}}}

The whole file, one panel per line. Save it as good.json:

{
 "title": "Podinfo - service health",
 "uid": "podinfo-good",
 "time": {"from": "now-15m", "to": "now"},
 "refresh": "10s",
 "templating": {"list": [
  {"name": "namespace", "label": "Namespace", "type": "query", "datasource": {"type": "prometheus", "uid": "prometheus"}, "query": {"query": "label_values(http_request_duration_seconds_count, namespace)", "refId": "ns"}, "refresh": 1, "current": {"text": "shop", "value": "shop"}},
  {"name": "pod", "label": "Pod", "type": "query", "datasource": {"type": "prometheus", "uid": "prometheus"}, "query": {"query": "label_values(http_request_duration_seconds_count{namespace=\"$namespace\"}, pod)", "refId": "pod"}, "refresh": 2, "includeAll": true, "multi": true, "allValue": ".*", "current": {"text": "All", "value": "$__all"}}
 ]},
 "panels": [
  {"type": "row", "title": "Is it working for users? (requests, errors, duration)", "gridPos": {"x": 0, "y": 0, "w": 24, "h": 1}, "collapsed": false, "panels": []},
  {"type": "stat", "title": "Requests per second", "gridPos": {"x": 0, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "reqps"}}, "description": "All requests the app answered, per second."},
  {"type": "stat", "title": "Errors (5xx share)", "gridPos": {"x": 6, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "100 * (sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\", status=~\"5..\"}[$__rate_interval])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 5}]}}}, "description": "Share of answers that were server errors. Orange from 1 percent, red from 5 percent.", "options": {"colorMode": "background"}},
  {"type": "stat", "title": "Slowest 5 percent (p95)", "gridPos": {"x": 12, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "s", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.5}, {"color": "red", "value": 1}]}}}, "description": "95 of every 100 requests were faster than this. Orange from 0.5 s, red from 1 s.", "options": {"colorMode": "background"}},
  {"type": "stat", "title": "Pods ready", "gridPos": {"x": 18, "y": 1, "w": 6, "h": 5}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(kube_pod_status_ready{namespace=\"$namespace\", pod=~\"podinfo.*\", condition=\"true\"})", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "none"}}, "description": "Podinfo pods that pass their readiness check."},
  {"type": "timeseries", "title": "Requests per second by status", "gridPos": {"x": 0, "y": 6, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum by (status) (rate(http_request_duration_seconds_count{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval]))", "legendFormat": "{{status}}"}], "fieldConfig": {"defaults": {"unit": "reqps"}}, "description": "One line per HTTP status code."},
  {"type": "timeseries", "title": "Duration: p50, p95, p99", "gridPos": {"x": 12, "y": 6, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "histogram_quantile(0.5, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": "p50"}, {"refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": "p95"}, {"refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", pod=~\"$pod\"}[$__rate_interval])))", "legendFormat": "p99"}], "fieldConfig": {"defaults": {"unit": "s"}}, "description": "The typical request (p50) and the slow ones (p95, p99)."},
  {"type": "row", "title": "Is it healthy inside? (use and saturation)", "gridPos": {"x": 0, "y": 14, "w": 24, "h": 1}, "collapsed": false, "panels": []},
  {"type": "timeseries", "title": "CPU used, share of the request", "gridPos": {"x": 0, "y": 15, "w": 12, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "100 * sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\"}[$__rate_interval])) / sum by (pod) (kube_pod_container_resource_requests{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\", resource=\"cpu\"})", "legendFormat": "{{pod}}"}], "fieldConfig": {"defaults": {"unit": "percent"}}, "description": "Above 100 percent the pod uses more CPU than it asked for."},
  {"type": "timeseries", "title": "Memory in use", "gridPos": {"x": 12, "y": 15, "w": 8, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum by (pod) (container_memory_working_set_bytes{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\"})", "legendFormat": "{{pod}}"}], "fieldConfig": {"defaults": {"unit": "bytes"}}, "description": "Working set memory: what the kernel counts against the limit."},
  {"type": "stat", "title": "Restarts (last hour)", "gridPos": {"x": 20, "y": 15, "w": 4, "h": 8}, "datasource": {"type": "prometheus", "uid": "prometheus"}, "targets": [{"refId": "A", "expr": "sum(increase(kube_pod_container_status_restarts_total{namespace=\"$namespace\", pod=~\"$pod\", container=\"podinfo\"}[1h]))", "legendFormat": ""}], "fieldConfig": {"defaults": {"unit": "none", "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}}, "description": "Container restarts. Anything above 0 needs a look.", "options": {"colorMode": "background"}}
 ]
}

or vector(0) makes the Errors panel show 0 when there are no errors, instead of No data. We run the same review first, then load it:

# The rebuilt dashboard: two questions (working for users? healthy inside?), units, thresholds, a Pod picker
python3 lint.py good.json
kubectl -n monitoring create configmap dash-good --from-file=good.json > /dev/null && kubectl -n monitoring label configmap dash-good grafana_dashboard=1 > /dev/null
sleep 30; curl -s "http://localhost:3000/api/dashboards/uid/podinfo-good" | jq -r '"loaded: " + .dashboard.title + ", version " + (.dashboard.version | tostring) + ", " + (.dashboard.panels | length | tostring) + " panels, variables: " + ([.dashboard.templating.list[].name] | join(", "))'
dashboard: Podinfo - service health
  ok    Requests per second                  
  ok    Errors (5xx share)                   
  ok    Slowest 5 percent (p95)              
  ok    Pods ready                           
  ok    Requests per second by status        
  ok    Duration: p50, p95, p99              
  ok    CPU used, share of the request       
  ok    Memory in use                        
  ok    Restarts (last hour)                 
0 of 9 panels need work
loaded: Podinfo - service health, version 1, 11 panels, variables: namespace, pod
The rebuilt dashboard
Real Grafana screen: 4.40 req/s, errors 0 percent, p95 1.38 s in red, 2 pods ready; p50 near 0 and p99 above 2 s

Read it in 10 seconds: traffic is fine, no errors, but p95 is red at 1.38 s. The duration graph shows why: p50 (the typical request) is a few milliseconds, while p95 and p99 are above 1 second. That is our slow container calling /delay/1. The old dashboard's "Latency 0.0665" (an average) completely hid it.

Break it: a bad release

We change the load pod so it also calls /status/500, a page that always fails, as if a bad release went out. Then we read the Errors panel of both dashboards after 90 seconds:

# Break it: a bad release. The load pod now also calls /status/500, a page that always fails
kubectl -n shop set env deploy/load -c load ERROR_PATH=/status/500 > /dev/null && kubectl -n shop rollout status deploy/load --timeout=90s | tail -1
sleep 90
Q='100 * sum(rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m])) / sum(rate(http_request_duration_seconds_count{namespace="shop"}[1m]))'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '"good dashboard, Errors panel: " + (.data.result[0].value[1] | tonumber | round | tostring) + " percent (red from 5)"'
Q='sum(http_requests_total{namespace="shop",code="500"})'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '"bad dashboard, Errors panel: " + (if (.data.result | length) == 0 then "No data" else .data.result[0].value[1] end)'
📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
deployment "load" successfully rolled out
good dashboard, Errors panel: 47 percent (red from 5)
bad dashboard, Errors panel: No data
The rebuilt dashboard during the incident
Real Grafana screen: errors 32.7 percent in red, a new status 500 line, p95 drops to 4.97 ms
The first dashboard during the incident
Real Grafana screen: the first dashboard still shows Errors: No data during the incident

The rebuilt dashboard turns red at once and shows a new status 500 line. The first dashboard still says No data: on that dashboard the incident is invisible.

One more lesson in the screenshot: p95 got better (4.97 ms, green) during the incident. The failing calls answer instantly, so there are many more fast requests and the slow ones drop out of the top 5 percent. Never judge a release by latency alone; read Errors and Duration together. That is why RED puts them side by side.

Fix it: roll back

We roll the load pod back to its old settings and watch the same number the Errors panel shows:

# Fix it: roll the bad release back, then watch the Errors panel go back to 0
Q='100 * (sum(rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace="shop"}[1m]))'
kubectl -n shop set env deploy/load -c load ERROR_PATH- > /dev/null && kubectl -n shop rollout status deploy/load --timeout=90s | tail -1
for i in $(seq 1 12); do sleep 15; V=$(curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '.data.result[0].value[1] | tonumber | round'); echo "after $((i*15))s: errors $V percent"; [ "$V" = 0 ] && break; done
deployment "load" successfully rolled out
after 15s: errors 42 percent
after 30s: errors 37 percent
after 45s: errors 30 percent
after 60s: errors 22 percent
after 75s: errors 9 percent
after 90s: errors 0 percent
Break it and fix it in the terminal
Break it: errors 47 percent on the rebuilt dashboard, No data on the first; fix it: back to 0 percent

The share falls step by step because the panel looks at the last minute: after 90 seconds every request in the window is good again.

After the fix
Real Grafana screen after the rollback: errors back to 0 percent

Dashboards as code

Why: the JSON file in Git is the truth. Because Grafana loaded the dashboard from a file, it marks it provisioned: a change made by hand in the Grafana screen cannot be saved over it, and a reviewer can see every change in a pull request.

# Dashboards as code: the file in Git is the truth. A change made by hand in the Grafana UI is lost on the next load
curl -s "http://localhost:3000/api/dashboards/uid/podinfo-good" | jq '.dashboard | del(.id, .version, .iteration)' > live.json
python3 - <<'P'
import json
a, b = json.load(open('good.json')), json.load(open('live.json'))
same = [p['title'] for p in a['panels']] == [p['title'] for p in b['panels']]
print('panels in Grafana match the file in Git:', same)
P
curl -s "http://localhost:3000/api/dashboards/uid/podinfo-good" | jq -r '"provisioned from a file (read-only in the UI): " + (.meta.provisioned | tostring)'
panels in Grafana match the file in Git: True
provisioned from a file (read-only in the UI): true

POC: prove it works

  • The review script failed the first dashboard on all 5 panels and passed the rebuilt one on all 9 (Steps 3 and 4).
  • During the bad release, the rebuilt dashboard showed 47 percent errors in red while the first one showed No data (Break it).
  • After the rollback the errors went back to 0 percent within 90 seconds (Fix it).
  • Grafana reports the dashboard as provisioned from the file, matching Git (Dashboards as code).

Clean up (delete everything)

Why: the stack is free only while the cluster runs, and the cluster is the cost. Helm leaves the Prometheus Operator's custom types behind on purpose, so we remove them too.

# Clean up: the stack, the app and the Prometheus Operator types
pkill -f "[p]ort-forward svc/monitoring"
helm uninstall monitoring -n monitoring --wait 2>&1 | tail -1
kubectl delete namespace shop monitoring --wait=true 2>&1 | tail -2
kubectl get crd -o name | grep monitoring.coreos.com | xargs -r kubectl delete > /dev/null && echo "Prometheus Operator types removed"
echo "namespaces left from this lab: $(kubectl get ns shop monitoring --no-headers 2>/dev/null | wc -l)"
rm -f ~/app.yaml ~/values.yaml ~/servicemonitor.yaml ~/bad.json ~/good.json ~/lint.py ~/live.json ~/pf-prom.log ~/pf-grafana.log   # prints nothing
release "monitoring" uninstalled
namespace "shop" deleted
namespace "monitoring" deleted
Prometheus Operator types removed
namespaces left from this lab: 0
Clean up
Provisioned from the file, then 0 lab namespaces left

When you have finished all your Kubernetes labs, delete the cluster:

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet

FAQ

How many panels should a dashboard have? As few as answer its question. Our service dashboard has 9. Put deep-dive panels (per-path latency, garbage collection, queue sizes) on a second dashboard and link to it.

RED or USE? Both, for different things. RED tells you whether users are hurting; USE tells you whether a resource (CPU, memory, disk) is the reason.

Why $__rate_interval and not [1m]? Grafana sets it from the scrape interval and the time range, so the graph stays smooth when you zoom out to 7 days. In our review script we replaced it with 1m.

Why not an average latency? An average mixes many fast requests with a few slow ones and shows a number nobody experienced. Our average was 0.0665 s while 1 in 20 requests took more than 1.3 s.

Can I still edit the dashboard in the screen? Yes, to try things. Then copy the JSON (Share > Export) into Git and let provisioning load it.

Troubleshooting

You seeWhyFix
A panel shows No dataWrong metric name or label, or no trafficRun the query in Prometheus first; add or vector(0) where 0 is a real answer
A graph only climbsA counter drawn without rate()Wrap it: sum(rate(metric[$__rate_interval]))
A legend as long as the screenEvery label printedAggregate (sum by (status)) and set legendFormat
The dashboard does not appearThe ConfigMap has no grafana_dashboard=1 label or is in another namespaceLabel it and keep it in the namespace the sidecar watches
Grafana says the dashboard cannot be savedIt is provisioned from a fileChange the file in Git and apply it again
The Pod drop-down is emptyThe variable query has no data for that namespaceCheck the Namespace drop-down first

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Prometheus and Grafana on Kubernetes from Scratch⏱ 17 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →