CareerByteCode
ByteBlogby CareerByteCode

Chaos Engineering on Kubernetes with Chaos Mesh

What you build: Chaos Mesh on GKE killing pods and slowing the network, a weak setting found and fixed with real success rates. Every step tested.

Chaos Engineering on Kubernetes with Chaos Mesh

✍️ Translate this blog into Hindi, Tamil or Telugu

Every system fails sometimes: pods crash, nodes disappear, networks get slow. Chaos engineering means causing those failures on purpose, in a controlled way, to learn how your system behaves before a real outage teaches you. Chaos Mesh is an open-source tool that does it on Kubernetes with simple YAML. In this lab you kill pods on a schedule, find the setting that makes users notice, fix it, and add network delay to see how users feel slowness.

Who it is for: SRE and DevOps beginners who know Deployments and Services. Time: about 1 hour. Cost: a small GKE cluster for about 15 minutes, covered by the free trial credit. Delete it at the end.

Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl and helm); Cloud Shell in your browser has the same tools.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: chaos engineering in plain English

A chaos experiment has four parts:

  1. Steady state: a number that says "users are fine", measured before you start. Ours: the share of successful calls.
  2. Hypothesis: what you expect, for example "with 3 replicas, users barely notice a pod being killed every 15 seconds".
  3. Experiment: inject one failure, in a small and limited scope (one namespace, one app, a fixed time).
  4. Learn: compare with the steady state. If users noticed, fix the system, then run the experiment again.
How a Chaos Mesh experiment works
A chaos object selects pods by label; the controller and the per-node daemon inject the failure; a prober measures what users feel

Words you will use:

  • Chaos Mesh controller: watches chaos objects and decides what to break. The chaos daemon runs on every node and does the breaking (kill a container, slow the network).
  • PodChaos: kill or fail pods. NetworkChaos: delay, drop or corrupt network traffic.
  • Schedule: repeats an experiment, for example "every 15 seconds".
  • Blast radius: how much can break. Ours is one namespace and one app label.
  • PodDisruptionBudget (PDB): tells Kubernetes how many pods must stay up during planned evictions such as node upgrades.
  • Real-life example: a fire drill. You do not wait for a real fire to find out the exit door is locked.

Before you start

  1. A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
  2. One small GKE cluster. We created it once for this batch of Kubernetes labs:
gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --enable-dataplane-v2

Part B: the lab

Step 1: install Chaos Mesh

Why: GKE nodes run containers with containerd, so the chaos daemon needs containerd's socket path. We also skip the web dashboard to keep the install small.

# Install Chaos Mesh with Helm (GKE nodes use containerd, so the chaos daemon needs its socket path)
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
helm repo add chaos-mesh https://charts.chaos-mesh.org > /dev/null 2>&1 && helm repo update chaos-mesh > /dev/null 2>&1
helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --create-namespace --version 2.7.1 \
  --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock --set dashboard.create=false --wait --timeout 6m 2>&1 | grep -E "STATUS|REVISION"
kubectl -n chaos-mesh get pods --no-headers | awk '{print $1, $3}'
STATUS: deployed
REVISION: 1
chaos-controller-manager-9cbbf6d96-k4gs9 Running
chaos-controller-manager-9cbbf6d96-pbqkr Running
chaos-controller-manager-9cbbf6d96-pxzsz Running
chaos-daemon-4jvfd Running
chaos-daemon-wnhhv Running
chaos-dns-server-5d57657555-pjkdg Running

Three controller copies (for high availability), one daemon per node (we have two nodes), and a small DNS server Chaos Mesh uses for DNS experiments.

Step 2: the app and a prober (the steady state)

Why: you cannot see damage without a measurement. The app is a tiny echo server with 3 replicas behind a Service, with a readiness probe. The prober pod calls it twice a second and logs the time, the HTTP code and the response time. Save this as app.yaml:

# The app under test (3 replicas behind a Service) and a prober that calls it twice a second and logs: time, HTTP code, seconds
apiVersion: apps/v1
kind: Deployment
metadata: {name: web, namespace: chaos-demo}
spec:
  replicas: 3
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: hashicorp/http-echo:1.0
        args: ["-text=ok", "-listen=:5678"]
        readinessProbe: {httpGet: {path: /, port: 5678}, periodSeconds: 2}
        resources: {requests: {cpu: 10m, memory: 16Mi}}
---
apiVersion: v1
kind: Service
metadata: {name: web, namespace: chaos-demo}
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: 5678}]
---
apiVersion: v1
kind: Pod
metadata: {name: prober, namespace: chaos-demo}
spec:
  containers:
  - name: prober
    image: curlimages/curl:8.11.1
    command: ["sh", "-c", "while true; do echo \"$(date +%s) $(curl -s -o /dev/null -w '%{http_code} %{time_total}' --max-time 2 http://web/)\"; sleep 0.5; done"]
# The app (3 replicas) and a prober that calls it twice a second
kubectl create namespace chaos-demo > /dev/null && kubectl apply -f app.yaml > /dev/null && kubectl -n chaos-demo rollout status deploy/web --timeout=120s && kubectl -n chaos-demo wait pod/prober --for=condition=Ready --timeout=90s > /dev/null
sleep 30; echo "baseline, last 30 s:"; kubectl -n chaos-demo logs prober --since=30s | awk '{n++; if ($2 == 200) ok++; t += $3} END {printf "%d calls, %d ok (%.1f%%), average %.0f ms\n", n, ok, 100*ok/n, 1000*t/n}'
Waiting for deployment "web" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 2 of 3 updated replicas are available...
deployment "web" successfully rolled out
baseline, last 30 s:
58 calls, 58 ok (100.0%), average 3 ms
Install and steady state
Step 1 and 2: Chaos Mesh running, the app and its steady state: 58 of 58 calls ok

Steady state: 100 percent of calls succeed, in about 3 ms.

Step 3: Chaos Mesh checks who starts an experiment

Why: chaos tools can break production, so Chaos Mesh 2.7 runs in security mode by default: its webhook checks whether the person creating an experiment may do it. In our run, our Google Cloud login was refused even though it is a cluster admin through Google Cloud IAM. On a lab cluster that only you use, switching security mode off is fine. On a shared cluster, keep it on and give the people who run experiments Kubernetes RBAC roles, as the Chaos Mesh documentation describes (we did not get that to pass in our run).

# Chaos Mesh checks WHO starts an experiment (security mode, on by default). Our cloud login is refused here
kubectl apply -f pod-kill.yaml 2>&1 | cut -c1-150
# On a lab cluster that only you use, switch security mode off (keep it on in shared clusters and give people RBAC roles)
helm upgrade chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --version 2.7.1 --reuse-values --set dashboard.securityMode=false 2>&1 | grep -E "STATUS"
kubectl -n chaos-mesh rollout status deploy/chaos-controller-manager --timeout=180s | tail -1; sleep 10
kubectl apply -f pod-kill.yaml --dry-run=server 2>&1 | cut -c1-150
Error from server (Forbidden): error when creating "pod-kill.yaml": admission webhook "vauth.kb.io" denied the request: ACCOUNT_ID is forbi
STATUS: deployed
deployment "chaos-controller-manager" successfully rolled out
schedule.chaos-mesh.org/kill-web created (server dry run)
Security mode
Step 3: the webhook refuses the experiment, security mode off, the same request accepted

The first line is the refusal; the last line is the same request accepted after the change (a server dry run, so nothing was created yet).

Step 4: experiment 1, kill a pod every 15 seconds

Why: pods die in real life (node upgrades, crashes, evictions). Hypothesis: with 3 replicas, users barely notice. The Schedule runs a PodChaos every 15 seconds, killing one random app: web pod in our namespace. Save this as pod-kill.yaml:

# Chaos experiment: every 15 seconds, kill one random pod of the web app (a Schedule of PodChaos)
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata: {name: kill-web, namespace: chaos-demo}
spec:
  schedule: "@every 15s"
  historyLimit: 5
  concurrencyPolicy: Forbid
  type: PodChaos
  podChaos:
    action: pod-kill
    mode: one
    selector:
      namespaces: [chaos-demo]
      labelSelectors: {app: web}
# Experiment 1 (hypothesis: with 3 replicas, users barely notice a pod being killed every 15 s). Run it for 60 s
kubectl apply -f pod-kill.yaml && sleep 62
kubectl -n chaos-demo get podchaos --no-headers 2>/dev/null | wc -l | sed 's/$/ pod-kill runs by the schedule (it keeps the last 5)/'; kubectl delete -f pod-kill.yaml > /dev/null
kubectl -n chaos-demo logs prober --since=60s | awk '{n++; if ($2 == 200) ok++} END {printf "during the experiment: %d calls, %d ok (%.1f%%)\n", n, ok, 100*ok/n}'
schedule.chaos-mesh.org/kill-web created
4 pod-kill runs by the schedule (it keeps the last 5)
during the experiment: 117 calls, 116 ok (99.1%)

Four pods killed in one minute, and 116 of 117 calls still succeeded (99.1 percent). One call landed on a pod at the moment it died. The hypothesis held.

Break it: the same experiment with one replica

Many apps run one replica in development and "it works fine". We scale to 1 and repeat the experiment:

# Break it: the same experiment with only 1 replica (a common "it works on dev" setting)
kubectl -n chaos-demo scale deploy web --replicas=1 > /dev/null && sleep 15
kubectl apply -f pod-kill.yaml > /dev/null && sleep 62 && kubectl delete -f pod-kill.yaml > /dev/null
kubectl -n chaos-demo logs prober --since=60s | awk '{n++; if ($2 == 200) ok++; else bad++} END {printf "1 replica: %d calls, %d ok (%.1f%%), %d failed\n", n, ok, 100*ok/n, bad}'
1 replica: 116 calls, 102 ok (87.9%), 14 failed

14 of 116 calls failed (87.9 percent success). Each time the only pod died, nothing could answer until Kubernetes started a new one and its readiness probe passed. In a real app with a slower start, this would be worse.

Fix it: more replicas and a PodDisruptionBudget

We go back to 3 replicas and add a PodDisruptionBudget that keeps at least 2 pods during planned evictions. Then we run the same experiment again:

# Fix it: back to 3 replicas, and a PodDisruptionBudget so planned evictions keep at least 2 pods
kubectl -n chaos-demo scale deploy web --replicas=3 > /dev/null && kubectl -n chaos-demo rollout status deploy/web --timeout=120s > /dev/null
kubectl -n chaos-demo create pdb web --selector=app=web --min-available=2 > /dev/null && kubectl -n chaos-demo get pdb web --no-headers | awk '{print "pdb", $1, "min available", $2, "allowed disruptions", $4}'
kubectl apply -f pod-kill.yaml > /dev/null && sleep 62 && kubectl delete -f pod-kill.yaml > /dev/null
kubectl -n chaos-demo logs prober --since=60s | awk '{n++; if ($2 == 200) ok++} END {printf "3 replicas again: %d calls, %d ok (%.1f%%)\n", n, ok, 100*ok/n}'
📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
pdb web min available 2 allowed disruptions 1
3 replicas again: 117 calls, 116 ok (99.1%)

Back to 99.1 percent. Note what the PDB does and does not do: it protects against planned disruptions (node drains during upgrades). Chaos Mesh's pod kill is like a crash, which a PDB cannot stop. The replicas are what saved us here; the PDB protects the next node upgrade.

Break it and fix it
3 replicas: 99.1 percent; 1 replica: 87.9 percent and 14 failures; 3 replicas and a PDB: 99.1 percent again

Step 5: experiment 2, a slow network

Why: slowness hurts users as much as errors, and it hides better. We add 300 ms of delay (plus or minus 20 ms) to every web pod for 60 seconds. Save this as net-delay.yaml:

# Chaos experiment: add 300 ms of network delay to every web pod for 60 seconds
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: {name: slow-web, namespace: chaos-demo}
spec:
  action: delay
  mode: all
  selector:
    namespaces: [chaos-demo]
    labelSelectors: {app: web}
  delay: {latency: "300ms", jitter: "20ms"}
  duration: "60s"
# Experiment 2: 300 ms of network delay on every web pod for 60 s. What do users feel?
kubectl apply -f net-delay.yaml > /dev/null && sleep 45
kubectl -n chaos-demo logs prober --since=40s | awk '{n++; if ($2 == 200) ok++; t += $3} END {printf "with 300 ms delay: %d calls, %d ok, average %.0f ms\n", n, ok, 1000*t/n}'
kubectl -n chaos-demo get networkchaos slow-web -o jsonpath='{"experiment phase: "}{.status.experiment.desiredPhase}{"\n"}'
sleep 30; kubectl delete -f net-delay.yaml > /dev/null; sleep 15
kubectl -n chaos-demo logs prober --since=12s | awk '{n++; t += $3} END {printf "after the experiment: average %.0f ms\n", 1000*t/n}'
with 300 ms delay: 36 calls, 36 ok, average 607 ms
experiment phase: Run
after the experiment: average 3 ms

Every call still succeeded, so an error-rate alert would stay quiet, but the average response time went from 3 ms to 607 ms. Why 607 and not 300? A new HTTP connection needs several network round trips (the TCP handshake, then the request), and each crossing of the slowed pod's network pays the delay. After the experiment ended, responses were back to 3 ms. Lesson: alert on latency as well as errors.

POC: prove it works

  • Steady state measured first: 100 percent success, 3 ms (Step 2).
  • Pod kills with 3 replicas: 99.1 percent; with 1 replica: 87.9 percent; fixed back to 99.1 percent (Step 4, Break it, Fix it).
  • Network delay: 100 percent success but 607 ms on average, back to 3 ms after (Step 5).

Clean up (delete everything)

Why: chaos tools should never stay installed by accident. We delete the experiments, the app, Chaos Mesh and its object types.

# Clean up: the experiments, the app, and Chaos Mesh itself
kubectl delete -f pod-kill.yaml -f net-delay.yaml --ignore-not-found > /dev/null 2>&1
kubectl delete namespace chaos-demo --wait=true 2>&1 | tail -1
helm uninstall chaos-mesh -n chaos-mesh 2>&1 | tail -1; kubectl delete namespace chaos-mesh --wait=true 2>&1 | tail -1
kubectl get crd -o name | grep chaos-mesh.org | xargs -r kubectl delete > /dev/null 2>&1; echo "chaos CRDs left: $(kubectl get crd -o name | grep -c chaos-mesh.org)"
namespace "chaos-demo" deleted
release "chaos-mesh" uninstalled
namespace "chaos-mesh" deleted
chaos CRDs left: 0
Clean up
The network delay experiment, then 0 Chaos Mesh object types left

When you have finished all your Kubernetes labs, delete the cluster:

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet

FAQ

Is it safe to run chaos in production? Only after you have run it many times in staging, with a small blast radius, a measured steady state, a stop button (delete the chaos object) and people watching. Start in staging, like this lab.

Chaos Mesh or Litmus? Both are open-source CNCF projects. Chaos Mesh uses simple YAML objects per failure type; Litmus organises experiments in workflows with a hub of ready-made ones. We tested Chaos Mesh only.

What other failures can Chaos Mesh inject? CPU and memory stress, disk faults, DNS errors, clock skew and HTTP faults, among others. We tested pod kill and network delay.

How do I stop an experiment at once? Delete the chaos object (kubectl delete -f pod-kill.yaml). The schedule stops and no new failures start.

Why did security mode refuse a cluster admin? On GKE, your rights come from Google Cloud IAM, and Chaos Mesh's own check could not see them in our run. We switched the check off on this lab cluster only.

Troubleshooting

You seeWhyFix
admission webhook "vauth.kb.io" denied the requestChaos Mesh security mode refuses your userLab cluster: --set dashboard.securityMode=false; shared cluster: RBAC per the Chaos Mesh docs
Network chaos does nothingWrong container runtime settings for the daemonOn GKE use chaosDaemon.runtime=containerd and /run/containerd/containerd.sock
The experiment hits other appsThe selector is too wideAlways set namespaces and labelSelectors
Calls fail even with 3 replicasNo readiness probe, so traffic goes to pods that are not readyAdd a readiness probe
Chaos object types left after uninstallHelm keeps CRDsDelete the chaos-mesh.org CRDs, as in the clean-up

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →SRE Interview Questions Answered with Hands-On Labs⏱ 9 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →