CareerByteCode
ByteBlogby CareerByteCode

Prometheus and Grafana on Kubernetes from Scratch

What you build: Prometheus, Grafana and Alertmanager on GKE with your first dashboard and an alert that really fires and clears, in your own Google Cloud account. Every step tested.

Prometheus and Grafana on Kubernetes from Scratch

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you build a complete metrics stack on Kubernetes from scratch: Prometheus collects numbers from your app, Grafana draws them on your first dashboard, and Alertmanager fires an alert when something goes wrong. Then you break the app on purpose, watch the alert fire, find the cause with one query and watch the alert clear.

Who it is for: beginners who know what a pod and a service are and want to see monitoring work end to end. Time: about 1.5 to 2 hours. Cost: a small GKE cluster costs about USD 0.30 an hour and the free trial credit covers it; this lab used the cluster for about 15 minutes. Delete the cluster at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got, including the Grafana screens. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl and helm); Cloud Shell in your browser has the same tools ready.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: metrics, Prometheus and Grafana in plain English

A metric is a number measured over time: requests per second, errors, memory used. Your app publishes its metrics as plain text on a web page, usually /metrics. Prometheus visits that page every few seconds (it scrapes it) and stores every number with a time stamp. Grafana asks Prometheus questions and draws the answers. Alertmanager receives alerts from Prometheus and sends the email or chat message.

How the metrics stack works
The app publishes metrics, Prometheus scrapes and checks rules, Grafana draws, Alertmanager notifies

Words you will use:

  • Scrape: Prometheus reads /metrics of each target every 15 seconds. Real-life example: an electricity meter reader who visits every house on a round.
  • Time series: one metric with one set of labels, for example http_request_duration_seconds_count{status="200"}: a counter that only goes up.
  • Labels: tags on a metric (status, path, pod) that let you split and filter.
  • PromQL: Prometheus's query language. rate(x[1m]) turns an ever-growing counter into "per second over the last minute".
  • ServiceMonitor: a Kubernetes object that tells Prometheus which service to scrape. You add one per app instead of editing Prometheus config.
  • Alert rule: a PromQL condition plus a time (for: 1m). When it is true long enough, the alert goes from inactive to pending to firing.
  • Helm chart: a ready-made package of Kubernetes YAML. We install the community chart kube-prometheus-stack, which brings Prometheus, Grafana, Alertmanager, the Prometheus Operator, node-exporter and kube-state-metrics together.

Before you start

  1. A Google Cloud account (the free trial works) and a project with billing linked.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console). It has gcloud, kubectl and helm.
  3. Set your project and turn on the Kubernetes Engine API. Replace the text in quotes with your own project id:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com

Part B: the lab

Step 1: create a small GKE cluster and connect

Why: everything runs inside Kubernetes. Two e2-standard-4 nodes, one zone, no GPU. We used this same cluster for four labs in a row (Argo CD, this one, Istio and Ollama), so it was created once:

gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --workload-pool=$PROJECT_ID.svc.id.goog
NAME         LOCATION    STATUS   CURRENT_NODE_COUNT  CURRENT_MASTER_VERSION
lab-cluster  us-east1-b  RUNNING  2                   1.36.4-gke.2377000
NAME          MACHINE_TYPE   DISK_SIZE_GB  INITIAL_NODE_COUNT
default-pool  e2-standard-4  50            2
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b 2>&1 | tail -1
kubectl get nodes
kubeconfig entry generated for lab-cluster.
NAME                                         STATUS   ROLES    AGE   VERSION
gke-lab-cluster-default-pool-27d9254d-1nv1   Ready    <none>   18m   v1.36.4-gke.2377000
gke-lab-cluster-default-pool-27d9254d-rl98   Ready    <none>   18m   v1.36.4-gke.2377000

Step 2: deploy a sample app that makes metrics

Why: we need something to monitor. podinfo is a small open-source web app that publishes Prometheus metrics on /metrics, and it has a page /status/500 that always fails, which we use later to break things. A second pod, load, calls podinfo 5 times a second so the graphs have data. Save this as app.yaml:

# podinfo: a small web app that publishes Prometheus metrics on /metrics, plus a load pod that calls it all the time
apiVersion: apps/v1
kind: Deployment
metadata: {name: podinfo, labels: {app: podinfo}}
spec:
  replicas: 2
  selector: {matchLabels: {app: podinfo}}
  template:
    metadata: {labels: {app: podinfo}}
    spec:
      containers:
      - name: podinfo
        image: ghcr.io/stefanprodan/podinfo:6.7.1
        ports: [{name: http, containerPort: 9898}]
        resources: {requests: {cpu: 20m, memory: 32Mi}}
---
apiVersion: v1
kind: Service
metadata: {name: podinfo, labels: {app: podinfo}}
spec:
  selector: {app: podinfo}
  ports: [{name: http, port: 9898, targetPort: 9898}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: load}
spec:
  replicas: 1
  selector: {matchLabels: {app: load}}
  template:
    metadata: {labels: {app: load}}
    spec:
      containers:
      - name: load
        image: curlimages/curl:8.10.1
        env: [{name: ERROR_PATH, value: ""}]
        command: ["sh", "-c", "while true; do curl -s -o /dev/null http://podinfo:9898/; [ -n \"$ERROR_PATH\" ] && curl -s -o /dev/null http://podinfo:9898$ERROR_PATH; sleep 0.2; done"]
        resources: {requests: {cpu: 10m, memory: 16Mi}}
kubectl create namespace shop
kubectl apply -n shop -f app.yaml
kubectl -n shop rollout status deploy/podinfo --timeout=180s
kubectl -n shop get pods
# the app publishes its own metrics as plain text on /metrics
kubectl -n shop run peek --rm -i -q --restart=Never --image=curlimages/curl:8.10.1 -- \
  sh -c "curl -s http://podinfo:9898/metrics | grep '^http_request_duration_seconds_count' | head -3"
namespace/shop created
deployment.apps/podinfo created
service/podinfo created
deployment.apps/load created
Waiting for deployment "podinfo" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "podinfo" rollout to finish: 1 of 2 updated replicas are available...
deployment "podinfo" successfully rolled out
NAME                      READY   STATUS    RESTARTS   AGE
load-7cfbc5c6c8-frh8s     1/1     Running   0          3s
podinfo-68d7484f5-c7zlb   1/1     Running   0          3s
podinfo-68d7484f5-hpj74   1/1     Running   0          3s

The last command was meant to print three lines of podinfo's /metrics page from a short-lived test pod. In our run it printed nothing: the test pod finished before kubectl attached to it (a known timing race with kubectl run --rm -i). Step 4 reads the same metrics through Prometheus, so nothing depends on it. If you want to see the raw page yourself, run kubectl -n shop port-forward svc/podinfo 9898:9898 and open http://localhost:9898/metrics.

The cluster and the app
Steps 1 and 2: 2 nodes, 2 podinfo pods and the load pod running

Step 3: install Prometheus, Grafana and Alertmanager with Helm

Why: one Helm command installs the whole stack. The settings:

  • grafana.adminPassword: a random password we create and keep in a file. Never type a real password into a command you share.
  • serviceMonitorSelectorNilUsesHelmValues=false and ruleSelectorNilUsesHelmValues=false: let Prometheus pick up your ServiceMonitors and rules, not only the chart's own.
  • retention=6h: keep only 6 hours of data (it is a lab).
  • kubeControllerManager, kubeScheduler, kubeEtcd, kubeProxy off: on GKE, Google runs these parts and Prometheus cannot reach them, so their alerts would only be noise.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts > /dev/null
helm repo update > /dev/null
openssl rand -hex 10 > .grafana-pass   # a lab-only Grafana password, never shown
helm install monitoring prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace \
  --set grafana.adminPassword="$(cat .grafana-pass)" \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.retention=6h \
  --set kubeControllerManager.enabled=false --set kubeScheduler.enabled=false \
  --set kubeEtcd.enabled=false --set kubeProxy.enabled=false \
  --wait --timeout 10m | head -8
helm list -n monitoring
kubectl get pods -n monitoring
NAME: monitoring
LAST DEPLOYED: Wed Nov 18 14:59:54 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=monitoring"
NAME      	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART                       	APP VERSION
monitoring	monitoring	1       	2026-11-18 14:59:54.558323939 +0000 UTC	deployed	kube-prometheus-stack-93.4.0	v0.95.2    
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-monitoring-kube-prometheus-alertmanager-0   2/2     Running   0          84s
monitoring-grafana-5fb5c8b5c4-4f9ps                      3/3     Running   0          88s
monitoring-kube-prometheus-operator-7465fdcc58-j9g9c     1/1     Running   0          88s
monitoring-kube-state-metrics-599b97c458-ln82s           1/1     Running   0          88s
monitoring-prometheus-node-exporter-mdjk7                1/1     Running   0          88s
monitoring-prometheus-node-exporter-qdc6b                1/1     Running   0          89s
prometheus-monitoring-kube-prometheus-prometheus-0       2/2     Running   0          84s
The stack is installed
Step 3: kube-prometheus-stack 93.4.0, all monitoring pods Running

Step 4: tell Prometheus to scrape the app, then query it

Why: a ServiceMonitor says "scrape the service labelled app: podinfo, port http, path /metrics, every 15 seconds". Save it as servicemonitor.yaml:

# Tell Prometheus: scrape the podinfo service on port "http", path /metrics, every 15 seconds
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: {name: podinfo, namespace: shop}
spec:
  selector: {matchLabels: {app: podinfo}}
  endpoints: [{port: http, path: /metrics, interval: 15s}]

Then we open Prometheus on localhost:9090 with kubectl port-forward and ask its API two questions: are the targets up, and how many requests per second does podinfo serve, split by status code?

kubectl apply -f servicemonitor.yaml
nohup kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 > pf-prom.log 2>&1 &
# give Prometheus a minute to find the new target and scrape it a few times
sleep 75
curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.labels.namespace=="shop") | .labels.pod + "  " + .health'
Q='sum by (status) (rate(http_request_duration_seconds_count{namespace="shop"}[1m]))'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '.data.result[] | "status " + .metric.status + ": " + (.value[1] | tonumber | . * 100 | round / 100 | tostring) + " requests per second"'
servicemonitor.monitoring.coreos.com/podinfo created
podinfo-68d7484f5-hpj74  up
podinfo-68d7484f5-c7zlb  up
status 200: 5.04 requests per second

Both podinfo pods are up, and the app serves about 5 requests a second, all with status 200. That matches the load pod (one call every 0.2 seconds).

Step 5: your first Grafana dashboard

Why: numbers in a terminal are fine for one question; a dashboard shows everything at a glance. Grafana in this chart has a helper (the "sidecar") that loads every ConfigMap labelled grafana_dashboard: "1". So the dashboard is just YAML you can keep in Git. Save it as dashboard.yaml:

# A Grafana dashboard stored as a ConfigMap. Grafana's helper (the "sidecar") loads every ConfigMap labelled grafana_dashboard=1.
apiVersion: v1
kind: ConfigMap
metadata:
  name: podinfo-dashboard
  namespace: monitoring
  labels: {grafana_dashboard: "1"}
data:
  podinfo.json: |
    {"title": "Podinfo - my first dashboard", "uid": "podinfo-lab", "time": {"from": "now-15m", "to": "now"}, "refresh": "10s",
     "panels": [
      {"type": "timeseries", "title": "Requests per second (by status)", "gridPos": {"x": 0, "y": 0, "w": 12, "h": 9},
       "targets": [{"expr": "sum by (status) (rate(http_request_duration_seconds_count{namespace=\"shop\"}[1m]))", "legendFormat": "status {{status}}"}]},
      {"type": "stat", "title": "Error rate (5xx, %)", "gridPos": {"x": 12, "y": 0, "w": 6, "h": 9}, "fieldConfig": {"defaults": {"unit": "percent", "decimals": 1}},
       "targets": [{"expr": "100 * (sum(rate(http_request_duration_seconds_count{namespace=\"shop\",status=~\"5..\"}[1m])) or vector(0)) / sum(rate(http_request_duration_seconds_count{namespace=\"shop\"}[1m]))"}]},
      {"type": "stat", "title": "Pods up", "gridPos": {"x": 18, "y": 0, "w": 6, "h": 9},
       "targets": [{"expr": "sum(up{namespace=\"shop\"})"}]},
      {"type": "timeseries", "title": "Latency p95 (seconds)", "gridPos": {"x": 0, "y": 9, "w": 24, "h": 8}, "fieldConfig": {"defaults": {"unit": "s"}},
       "targets": [{"expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{namespace=\"shop\"}[1m])))", "legendFormat": "p95"}]}
     ]}

Four panels: requests per second by status, error rate in percent, pods up, and the 95th percentile latency (histogram_quantile(0.95, ...): 95 out of 100 requests are faster than this).

kubectl apply -f dashboard.yaml
nohup kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 > pf-grafana.log 2>&1 &
sleep 20
# ask Grafana (with the lab admin password) which dashboards it loaded
curl -s -u "admin:$(cat .grafana-pass)" "http://localhost:3000/api/search?query=podinfo" | jq -r '.[] | "dashboard: " + .title + "  (uid " + .uid + ")"'
curl -s -u "admin:$(cat .grafana-pass)" http://localhost:3000/api/datasources | jq -r '.[] | "data source: " + .name + " -> " + .type'
configmap/podinfo-dashboard created
dashboard: Podinfo - my first dashboard  (uid podinfo-lab)
data source: Alertmanager -> alertmanager
data source: Prometheus -> prometheus

To open Grafana in Cloud Shell, run kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80, click Web Preview > Change port > 3000, and sign in as admin with the password in .grafana-pass. Then open Dashboards > Podinfo - my first dashboard. You will see it with real data in the Break it section below.

Step 6: an alert rule

Why: nobody watches dashboards all night. An alert rule watches for you. Ours fires when more than 10 percent of requests fail with a 5xx error for 1 minute. Save it as alert-rule.yaml:

# Alert when more than 10% of requests fail with a 5xx error for 1 minute
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: {name: podinfo-alerts, namespace: shop}
spec:
  groups:
  - name: podinfo
    rules:
    - alert: PodinfoHighErrorRate
      expr: |
        sum(rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m]))
          / sum(rate(http_request_duration_seconds_count{namespace="shop"}[1m])) > 0.10
      for: 1m
      labels: {severity: warning}
      annotations:
        summary: "podinfo: more than 10% of requests fail"

The Prometheus Operator turns this into Prometheus config. That is not instant: in our first try we checked after 30 seconds and the rule was not there yet; in the second it was loaded but not yet checked (health unknown). So the step now waits until Prometheus reports health ok.

kubectl apply -f alert-rule.yaml
# the Prometheus Operator turns the rule into Prometheus config; wait until Prometheus has loaded and checked it once
for i in $(seq 1 30); do
  R=$(curl -s http://localhost:9090/api/v1/rules | jq -r '.data.groups[] | select(.name=="podinfo") | .rules[] | "rule " + .name + ": health " + .health + ", state " + .state')
  echo "$R" | grep -q "health ok" && break; sleep 10
done
echo "loaded after about $((i*10)) seconds"; echo "$R"
prometheusrule.monitoring.coreos.com/podinfo-alerts unchanged
loaded after about 10 seconds
rule PodinfoHighErrorRate: health ok, state inactive

state inactive is what you want: the rule works and nothing is wrong.

Break it: the app starts failing

We tell the load pod to also call /status/500 on every round, so about half of all requests fail. Then we watch the alert move through its states.

# Break it: the load pod now also calls /status/500, a page that always fails
kubectl -n shop set env deploy/load ERROR_PATH=/status/500
kubectl -n shop rollout status deploy/load --timeout=90s
for i in $(seq 1 24); do
  S=$(curl -s http://localhost:9090/api/v1/alerts | jq -r '[.data.alerts[] | select(.labels.alertname=="PodinfoHighErrorRate") | .state][0] // "none"')
  echo "after $((i*10))s: alert $S"; [ "$S" = "firing" ] && break; sleep 10
done
Q='100 * sum(rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m])) / sum(rate(http_request_duration_seconds_count{namespace="shop"}[1m]))'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '"error rate now: " + (.data.result[0].value[1] | tonumber | round | tostring) + " %"'
# the alert also reached Alertmanager, which would send the email or chat message
nohup kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093 > pf-am.log 2>&1 &
sleep 4; curl -s http://localhost:9093/api/v2/alerts | jq -r '.[] | select(.labels.alertname=="PodinfoHighErrorRate") | "Alertmanager has: " + .labels.alertname + " (" + .status.state + ") " + .annotations.summary'
📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
deployment.apps/load env updated
Waiting for deployment "load" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "load" rollout to finish: 1 old replicas are pending termination...
deployment "load" successfully rolled out
after 10s: alert none
after 20s: alert none
after 30s: alert none
after 40s: alert pending
after 50s: alert pending
after 60s: alert pending
after 70s: alert pending
after 80s: alert pending
after 90s: alert pending
after 100s: alert firing
error rate now: 49 %
Alertmanager has: PodinfoHighErrorRate (active) podinfo: more than 10% of requests fail
The alert fires
pending after about 40 s, firing after about 100 s; error rate 49 percent; Alertmanager has the alert

The rule became pending as soon as the error rate passed 10 percent, waited the full for: 1m, then went firing. Alertmanager received it; with a receiver configured (email, Slack, PagerDuty) it would now send the message. Here is the same moment in Grafana:

Grafana dashboard during the errors
Real Grafana screen: status 500 appears next to status 200; error rate 49.3 percent; 2 pods up

And Grafana's Alerting > Alert rules page shows our rule from Prometheus as firing (red):

Grafana alert rules
Real Grafana screen: PodinfoHighErrorRate firing, read from the Prometheus rule file

Find the cause and fix it

First question in any incident: which requests fail? PromQL can split the errors by the path label. podinfo names the route /status/{code} as status, so the answer path status, status 500 means "the /status page returns 500". The fix here is to stop calling the broken page; in real life it would be a code fix or a rollback.

# Find the cause: which page returns the 5xx errors?
Q='sum by (path, status) (rate(http_request_duration_seconds_count{namespace="shop",status=~"5.."}[1m]))'
curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$Q" | jq -r '.data.result[] | "failing: path " + .metric.path + " status " + .metric.status'
# Fix it: stop calling the broken page, then watch the alert clear
kubectl -n shop set env deploy/load ERROR_PATH-
kubectl -n shop rollout status deploy/load --timeout=90s
for i in $(seq 1 24); do
  S=$(curl -s http://localhost:9090/api/v1/alerts | jq -r '[.data.alerts[] | select(.labels.alertname=="PodinfoHighErrorRate") | .state][0] // "none"')
  echo "after $((i*10))s: alert $S"; [ "$S" = "none" ] && break; sleep 10
done
failing: path status status 500
deployment.apps/load env updated
Waiting for deployment "load" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "load" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "load" rollout to finish: 1 old replicas are pending termination...
deployment "load" successfully rolled out
after 10s: alert firing
after 20s: alert firing
after 30s: alert firing
after 40s: alert firing
after 50s: alert firing
after 60s: alert firing
after 70s: alert firing
after 80s: alert firing
after 90s: alert firing
after 100s: alert firing
after 110s: alert none
The alert clears
The failing path is found, the load pod is fixed, the alert clears after about 110 s

The alert stays firing for a while after the fix: the rule looks at a 1-minute window, so it needs a minute of good data first. Grafana shows the 500 line dropping to zero and the error rate back to 0.0 percent:

Grafana after the fix
Real Grafana screen: the status 500 line falls to 0 and the error rate shows 0.0 percent

POC: prove it works

  • Prometheus scrapes both podinfo pods (up) and answers PromQL (Step 4).
  • Grafana loaded the dashboard from a ConfigMap and draws live data (Step 5 and the screenshots).
  • The alert rule went inactive > pending > firing when errors passed 10 percent, reached Alertmanager, and cleared after the fix.

Clean up (delete everything)

Why: the stack and the app are free to keep only while the cluster runs, and the cluster is the cost. helm uninstall removes the stack, but Helm leaves the Prometheus Operator's custom types behind on purpose, so we delete them too.

pkill -f "[p]ort-forward svc/monitoring"
helm uninstall monitoring -n monitoring --wait | tail -1
kubectl delete namespace shop monitoring --wait=true
# Helm leaves the Prometheus Operator's custom types behind; remove them too
kubectl get crd -o name | grep monitoring.coreos.com | xargs -r kubectl delete > /dev/null && echo "Prometheus Operator types removed"
echo "namespaces left from this lab: $(kubectl get ns shop monitoring --no-headers 2>/dev/null | wc -l)"
echo "monitoring types left: $(kubectl get crd -o name | grep -c monitoring.coreos.com)"
release "monitoring" uninstalled
namespace "shop" deleted
namespace "monitoring" deleted
Prometheus Operator types removed
namespaces left from this lab: 0
monitoring types left: 0

If you are not doing another Kubernetes lab now, delete the cluster (we did this after our fourth lab on it):

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet 2>&1 | tail -1
echo "clusters left: $(gcloud container clusters list --format="value(name)" | wc -l)"
echo "cluster disks left: $(gcloud compute disks list --filter="name~^gke-lab-cluster OR name~^pvc-" --format="value(name)" | wc -l)"
echo "load balancers left: $(gcloud compute forwarding-rules list --format="value(name)" | wc -l)"
Deleted [https://container.googleapis.com/v1/projects/your-project-id/zones/us-east1-b/clusters/lab-cluster].
clusters left: 0
cluster disks left: 0
load balancers left: 0
Clean up
0 lab namespaces, 0 monitoring types, 0 clusters, 0 disks, 0 load balancers

FAQ

Google Cloud already has Managed Service for Prometheus. Why install my own? GKE can collect Prometheus metrics for you with Google Cloud Managed Service for Prometheus, and that is a good choice in production. This lab installs the open-source stack because it is what you meet in most companies and interviews, and it works the same on any Kubernetes.

Pull or push: how does Prometheus get the data? Prometheus pulls: it visits each target. Short jobs that end before a scrape can push to a Pushgateway instead.

What is the difference between a counter and a gauge? A counter only goes up (total requests); you use rate() on it. A gauge goes up and down (memory in use); you read it directly.

How do I get alert emails? Add a receiver to Alertmanager's config (email, Slack, PagerDuty, a webhook) through the chart's alertmanager.config values. We did not configure a receiver in this lab.

Why did the alert need 100 seconds to fire? The rule waits for: 1m after the condition is true, and the condition itself uses a 1-minute window. Shorter values react faster but give more false alarms.

Is the Grafana password safe? It is random, kept in a file and never printed. Grafana is reachable only through kubectl port-forward, not from the internet.

Troubleshooting

You seeWhyFix
Your target is missing in PrometheusThe ServiceMonitor labels or port name do not match the serviceMatch selector.matchLabels and the port name exactly; wait a minute
The rule is not listed yetThe Operator needs time to reload PrometheusWait up to a minute and check again (Step 6 loops)
Rule health unknownLoaded but not evaluated yetWait one evaluation interval (30 s by default)
Alerts for the controller manager or schedulerGKE manages these and Prometheus cannot reach themInstall with kubeControllerManager.enabled=false and kubeScheduler.enabled=false
A kubectl run --rm -i test prints nothingThe pod finished before kubectl attachedUse kubectl logs on a normal pod, or port-forward
Grafana dashboard does not appearConfigMap without the label or in another namespaceLabel grafana_dashboard: "1", namespace monitoring
Error: INSTALLATION FAILED: ... timed outNodes too small or busyUse 2 nodes of e2-standard-4, or raise --timeout

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Grafana Dashboards That Engineers Actually Use⏱ 15 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →