Istio Service Mesh for Beginners on Kubernetes
What you build: Istio on GKE with a 90/10 canary split, strict mTLS that refuses outsiders, and retries that hide a flaky service, in your own Google Cloud account. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
In this lab you add the Istio service mesh to a Kubernetes cluster and use it for three things every platform team needs: traffic splitting for a safe canary release (90/10, then 50/50), mutual TLS (mTLS) so only trusted workloads can talk to each other, and automatic retries for a flaky service. You never change the app's code: Istio does all of it from the outside.
Who it is for: people who know basic Kubernetes (pods, deployments, services) and keep hearing "service mesh" in job posts. Time: about 1.5 to 2 hours. Cost: a small GKE cluster costs about USD 0.30 an hour and the free trial credit covers it; this lab used the cluster for about 12 minutes. Delete the cluster at the end.
Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud and kubectl); Cloud Shell in your browser has the same tools ready.
Part A: a service mesh in plain English
In a microservices app, services call each other over the network all the time. Every team then needs the same things: encryption, retries, timeouts, canary releases and metrics. Without a mesh, each team writes that code again in each language. A service mesh moves it out of the app and into the network layer.

Words you will use:
- Sidecar: Istio adds a second container, the Envoy proxy, to every pod in the mesh. All traffic in and out of your app goes through it. Real-life example: a personal assistant who handles all your calls: answers, encrypts, tries again when the line is busy.
- Control plane (istiod): gives every sidecar its rules and its certificate. The sidecars together are the data plane.
- VirtualService: a routing rule: "for calls to
web, send 90 percent here and 10 percent there". - DestinationRule: names the groups of pods behind a service (subsets, for example
v1andv2by label). - Canary release: send a small share of real users to the new version first, watch it, then move more.
- mTLS (mutual TLS): both sides show a certificate and the traffic is encrypted. PeerAuthentication STRICT = refuse anything that is not mTLS.
- Retry: when a call fails with a 5xx error, the sidecar quietly tries again.
Before you start
- A Google Cloud account (the free trial works) and a project with billing linked.
- Open Cloud Shell (the
>_icon at the top right of the Google Cloud console). - Set your project and turn on the Kubernetes Engine API. Replace the text in quotes with your own project id:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.comPart B: the lab
Step 1: create a small GKE cluster and connect
Why: Istio runs inside Kubernetes. Two e2-standard-4 nodes, one zone, no GPU. We used this same cluster for four labs in a row, so it was created once:
gcloud container clusters create lab-cluster \
--zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
--disk-type=pd-standard --disk-size=50 --release-channel=regular \
--no-enable-autoupgrade --workload-pool=$PROJECT_ID.svc.id.googNAME LOCATION STATUS CURRENT_NODE_COUNT CURRENT_MASTER_VERSION
lab-cluster us-east1-b RUNNING 2 1.36.4-gke.2377000
NAME MACHINE_TYPE DISK_SIZE_GB INITIAL_NODE_COUNT
default-pool e2-standard-4 50 2gcloud container clusters get-credentials lab-cluster --zone=us-east1-b 2>&1 | tail -1
kubectl get nodeskubeconfig entry generated for lab-cluster.
NAME STATUS ROLES AGE VERSION
gke-lab-cluster-default-pool-27d9254d-1nv1 Ready <none> 33m v1.36.4-gke.2377000
gke-lab-cluster-default-pool-27d9254d-rl98 Ready <none> 33m v1.36.4-gke.2377000Step 2: install istioctl and Istio
Why: istioctl is Istio's command line tool; the download script fetches the newest release. The minimal profile installs only the control plane (istiod), which is all you need for traffic rules and mTLS inside the cluster. (The demo and default profiles also add gateways for traffic from the internet.)
curl -sL https://istio.io/downloadIstio | sh - > /dev/null 2>&1
cd istio-*/ && sudo install -m 555 bin/istioctl /usr/local/bin/istioctl && cd ..
istioctl version --remote=false
# "minimal" = only the control plane (istiod): enough for traffic rules and mTLS inside the cluster
istioctl install --set profile=minimal -y 2>&1 | tail -3
kubectl -n istio-system get podsclient version: 1.32.0
โ CNI installed ๐ชข
- Pruning removed resources
โ Installation completeNAME READY STATUS RESTARTS AGE
istiod-7d4f59984d-hqvbp 1/1 Running 0 21s
Step 3: deploy two versions, a flaky service and a client
Why: to see traffic splitting we need two versions of one service. web-v1 answers "web v1", web-v2 answers "web v2 (new)"; both carry the label app: web and one version label, and one Service web covers both. orders is go-httpbin, a test server whose page /status/200,503 answers 200 or 503 at random: our flaky service. client is a pod we send requests from. Save this as apps.yaml:
# Two versions of the same "web" service, a flaky "orders" service and a client pod to send requests from
apiVersion: apps/v1
kind: Deployment
metadata: {name: web-v1}
spec:
selector: {matchLabels: {app: web, version: v1}}
template:
metadata: {labels: {app: web, version: v1}}
spec:
containers:
- {name: web, image: "hashicorp/http-echo:1.0", args: ["-text=web v1", "-listen=:5678"], ports: [{containerPort: 5678}], resources: {requests: {cpu: 10m, memory: 16Mi}}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web-v2}
spec:
selector: {matchLabels: {app: web, version: v2}}
template:
metadata: {labels: {app: web, version: v2}}
spec:
containers:
- {name: web, image: "hashicorp/http-echo:1.0", args: ["-text=web v2 (new)", "-listen=:5678"], ports: [{containerPort: 5678}], resources: {requests: {cpu: 10m, memory: 16Mi}}}
---
apiVersion: v1
kind: Service
metadata: {name: web}
spec:
selector: {app: web}
ports: [{name: http, port: 80, targetPort: 5678}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: orders}
spec:
selector: {matchLabels: {app: orders}}
template:
metadata: {labels: {app: orders}}
spec:
containers:
- {name: orders, image: "mccutchen/go-httpbin:v2.15.0", ports: [{containerPort: 8080}], resources: {requests: {cpu: 10m, memory: 16Mi}}}
---
apiVersion: v1
kind: Service
metadata: {name: orders}
spec:
selector: {app: orders}
ports: [{name: http, port: 80, targetPort: 8080}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: client}
spec:
selector: {matchLabels: {app: client}}
template:
metadata: {labels: {app: client}}
spec:
containers:
- {name: client, image: "curlimages/curl:8.10.1", command: ["sleep", "infinity"], resources: {requests: {cpu: 10m, memory: 16Mi}}}The label istio-injection=enabled on the namespace tells Istio to add the sidecar to every new pod there.
kubectl create namespace mesh
kubectl label namespace mesh istio-injection=enabled
kubectl apply -n mesh -f apps.yaml
kubectl -n mesh rollout status deploy/web-v1 deploy/web-v2 deploy/orders deploy/client --timeout=180s > /dev/null
# READY 2/2 = your app container + the Envoy sidecar that Istio added
kubectl -n mesh get podsnamespace/mesh created
namespace/mesh labeled
deployment.apps/web-v1 created
deployment.apps/web-v2 created
service/web created
deployment.apps/orders created
service/orders created
deployment.apps/client created
NAME READY STATUS RESTARTS AGE
client-bcfbbf867-lsl5g 2/2 Running 0 34s
orders-5d6947bfb4-w789r 2/2 Running 0 34s
web-v1-867d87f445-zf4fl 2/2 Running 0 35s
web-v2-7668659586-8677t 2/2 Running 0 35sREADY 2/2 on every pod: your container plus the Envoy sidecar.
Step 4: a canary release with a 90/10 traffic split
Why: first we send 100 requests with no Istio rule: Kubernetes spreads them over both versions, roughly half and half. That is a problem when v2 is new and maybe buggy. Then we add the rule. Save this as split-90-10.yaml:
# DestinationRule: name the two versions (subsets). VirtualService: send 90% to v1 and 10% to v2.
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: {name: web}
spec:
host: web
subsets:
- {name: v1, labels: {version: v1}}
- {name: v2, labels: {version: v2}}
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: {name: web}
spec:
hosts: [web]
http:
- route:
- {destination: {host: web, subset: v1}, weight: 90}
- {destination: {host: web, subset: v2}, weight: 10}# before any rule: Kubernetes spreads calls over all web pods (about 50/50)
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 100); do curl -s http://web; done' | sort | uniq -c
kubectl apply -n mesh -f split-90-10.yaml
sleep 5
echo "--- with the 90/10 rule ---"
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 100); do curl -s http://web; done' | sort | uniq -c
istioctl analyze -n mesh 46 web v1
54 web v2 (new)
destinationrule.networking.istio.io/web created
virtualservice.networking.istio.io/web created
--- with the 90/10 rule ---
91 web v1
9 web v2 (new)
โ No validation issues found when analyzing namespace: mesh.
46/54 without the rule, 91/9 with it. The split is by chance per request, so it is close to 90/10, not exact. istioctl analyze checks your Istio config for mistakes (wrong host names, subsets that match no pods) and found none.
Step 5: move the canary to 50/50
Why: v2 looks fine, so we give it more traffic. This is just two numbers in the VirtualService, with no new pods and no restart. In real life you would watch the error rate and latency between each move.
# the canary looks fine: move to 50/50 by changing two numbers
kubectl -n mesh patch virtualservice web --type=json -p '[{"op":"replace","path":"/spec/http/0/route/0/weight","value":50},{"op":"replace","path":"/spec/http/0/route/1/weight","value":50}]'
sleep 5
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 100); do curl -s http://web; done' | sort | uniq -cvirtualservice.networking.istio.io/web patched
49 web v1
51 web v2 (new)
Break it and fix it: strict mTLS
We tell Istio to accept only mTLS traffic in the mesh namespace. Save this as mtls-strict.yaml:
# Only accept mTLS traffic (encrypted, with a workload certificate) in namespace "mesh"
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: mesh}
spec:
mtls: {mode: STRICT}Then a pod outside the mesh (namespace outside, no sidecar, so no certificate) calls web, and a pod inside calls it too:
kubectl apply -f mtls-strict.yaml
# Break it: a pod OUTSIDE the mesh (no sidecar, no certificate) calls web
kubectl create namespace outside
kubectl -n outside run outsider --image=curlimages/curl:8.10.1 --restart=Never -- sleep 3600
kubectl -n outside wait pod/outsider --for=condition=Ready --timeout=90s > /dev/null
sleep 5
kubectl -n outside exec outsider -- curl -s -m 5 http://web.mesh; echo "exit code from outside: $?"
# a pod inside the mesh still works (its sidecar does mTLS for it)
kubectl -n mesh exec deploy/client -- curl -s -m 5 http://web; echo "exit code from inside: $?"peerauthentication.security.istio.io/default created
namespace/outside created
pod/outsider created
command terminated with exit code 56
exit code from outside: 56
web v1
exit code from inside: 0
The outsider's call fails with curl exit code 56 ("failure receiving data": the sidecar on the web side closed the plain-text connection). The client inside the mesh works, because its sidecar does mTLS for it. This is exactly what you want in production: a stolen or unknown workload cannot talk to your services. Fix: if the outsider is a real part of your system, bring it into the mesh.
# Fix it: bring the outside pod into the mesh (label the namespace, start the pod again)
kubectl label namespace outside istio-injection=enabled
kubectl -n outside delete pod outsider --wait=true > /dev/null
kubectl -n outside run outsider --image=curlimages/curl:8.10.1 --restart=Never -- sleep 3600
kubectl -n outside wait pod/outsider --for=condition=Ready --timeout=90s > /dev/null
kubectl -n outside get pod outsider
kubectl -n outside exec outsider -c outsider -- curl -s -m 5 http://web.mesh; echo "exit code: $?"📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →namespace/outside labeled
pod/outsider created
NAME READY STATUS RESTARTS AGE
outsider 2/2 Running 0 3s
web v2 (new)
exit code: 0
The pod now has a sidecar (2/2), gets a certificate from istiod, and its call works.
Retries for a flaky service
Our orders service fails about half the time. Save this rule as orders-retries.yaml:
# Retry a failed call to "orders" up to 3 times when it answers 5xx
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: {name: orders}
spec:
hosts: [orders]
http:
- route: [{destination: {host: orders}}]
retries: {attempts: 3, perTryTimeout: 2s, retryOn: "5xx"}We send 40 calls before and 40 after the rule:
# orders/status/200,503 answers 200 or 503 at random: a flaky service
echo "without retries:"
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code}\n" http://orders/status/200,503; done' | sort | uniq -c
kubectl apply -n mesh -f orders-retries.yaml
sleep 5
echo "with 3 retries on 5xx:"
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code}\n" http://orders/status/200,503; done' | sort | uniq -cwithout retries:
19 200
21 503
virtualservice.networking.istio.io/orders created
with 3 retries on 5xx:
36 200
4 503
From 21 failures to 4, with no code change. Why not 0? Each try has about a 1 in 2 chance to fail, so all 4 tries (1 call plus 3 retries) fail about 1 time in 16: on 40 calls that is about 2 to 3. Retries hide short glitches, not a broken service, and they add load, so keep attempts small and only retry calls that are safe to repeat.
POC: prove it works
- Traffic split: 91/9 with the 90/10 rule, 49/51 after the move to 50/50.
- mTLS STRICT: a pod without a sidecar is refused (exit 56); the same pod inside the mesh works.
- Retries: failed calls dropped from 21 of 40 to 4 of 40.
istioctl analyze: no validation issues.
Clean up (delete everything)
Why: remove the apps, Istio and its custom types, so the cluster is clean for the next lab or for deletion.
kubectl delete namespace mesh outside --wait=true
istioctl uninstall --purge -y 2>&1 | tail -1
kubectl delete namespace istio-system --wait=true
echo "namespaces left from this lab: $(kubectl get ns mesh outside istio-system --no-headers 2>/dev/null | wc -l)"
echo "Istio types left: $(kubectl get crd -o name | grep -c istio.io)"namespace "mesh" deleted
namespace "outside" deleted
โ Uninstall completenamespace "istio-system" deleted
namespaces left from this lab: 0
Istio types left: 0If you are not doing another Kubernetes lab now, delete the cluster (we did this after our fourth lab on it):
gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet 2>&1 | tail -1
echo "clusters left: $(gcloud container clusters list --format="value(name)" | wc -l)"
echo "cluster disks left: $(gcloud compute disks list --filter="name~^gke-lab-cluster OR name~^pvc-" --format="value(name)" | wc -l)"
echo "load balancers left: $(gcloud compute forwarding-rules list --format="value(name)" | wc -l)"Deleted [https://container.googleapis.com/v1/projects/your-project-id/zones/us-east1-b/clusters/lab-cluster].
clusters left: 0
cluster disks left: 0
load balancers left: 0
FAQ
Do I need a service mesh? Not for 3 services. It pays off when many teams run many services and need the same security, traffic control and visibility everywhere. It also adds CPU and memory (one sidecar per pod) and something new to learn and run.
What about Istio ambient mode? Istio also has an "ambient" mode without sidecars (a per-node proxy plus optional waypoint proxies). This lab uses the classic sidecar mode, which is what most guides and interviews still describe.
Is GKE's own Cloud Service Mesh the same thing? Cloud Service Mesh is Google's managed offering built on Istio's APIs: the same VirtualService and DestinationRule ideas, with Google running the control plane. We tested open-source Istio only.
How do users from the internet reach a mesh service? Through an Istio ingress gateway (or the Kubernetes Gateway API). The minimal profile we used does not install one.
Why did the split show 91/9 and not exactly 90/10? Each request is routed by chance with those weights. Over thousands of requests it gets very close to 90/10.
Where do I see mesh metrics? Every sidecar publishes Prometheus metrics (requests, errors, latency per service). Add Prometheus and Grafana or Kiali to see them.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
Pods show 1/1, not 2/2 | The namespace has no istio-injection=enabled label, or the pods started before it | Label the namespace, then kubectl rollout restart deploy -n <ns> |
| The traffic split is ignored | No DestinationRule subsets, or labels do not match | Check version labels; run istioctl analyze |
curl exit code 56 between services | STRICT mTLS and the caller has no sidecar | Bring the caller into the mesh, or use PERMISSIVE while migrating |
| Retries do not help | The service fails every time, or the error is not 5xx | Fix the service; check retryOn |
istioctl install hangs | Nodes too small | Use e2-standard-4 nodes or the minimal profile |
| Exec into a pod asks "Defaulted container" | The pod has two containers now | Add -c <container name> |
What next
- Learn how the network below the mesh works: Google Cloud VPC for Beginners.
- New to Kubernetes? Start with Zero to Kubernetes in 2.5 Hours.
- Watch traces across services: OpenTelemetry for Beginners.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamNetworking Interview Questions for Cloud Engineers, Proved in Labs
What you build: 10 cloud networking interview answers, each proved on Google Cloud: CIDR, Cloud NAT, firewall priority, DNS, TLS and routes. Every step tested.
#advanced
CareerByteCode TeamLinkerd vs Istio: Lightweight Service Mesh Hands-On
What you build: Istio and Linkerd on one GKE cluster, the same app and load measured side by side, and a plain-text caller test. Every step tested.
#advanced
CareerByteCode TeamCilium CNI: eBPF Networking and Security for Kubernetes
What you build: Cilium network policies on GKE Dataplane V2: default deny, dropped packets seen in the kernel, egress control and a DNS mistake fixed. Every step tested.
#beginners
CareerByteCode TeamGoogle Cloud VPC for Beginners: Subnets, Routes and Firewalls
What you build: a custom VPC with two subnets and firewall rules, in your own Google Cloud account. Every step, command and screenshot explained for beginners.
💬 Comments (0)
Top · NewestSign in to comment