CareerByteCode
ByteBlogby CareerByteCode

Istio Service Mesh for Beginners on Kubernetes

What you build: Istio on GKE with a 90/10 canary split, strict mTLS that refuses outsiders, and retries that hide a flaky service, in your own Google Cloud account. Every step tested.

Istio Service Mesh for Beginners on Kubernetes

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you add the Istio service mesh to a Kubernetes cluster and use it for three things every platform team needs: traffic splitting for a safe canary release (90/10, then 50/50), mutual TLS (mTLS) so only trusted workloads can talk to each other, and automatic retries for a flaky service. You never change the app's code: Istio does all of it from the outside.

Who it is for: people who know basic Kubernetes (pods, deployments, services) and keep hearing "service mesh" in job posts. Time: about 1.5 to 2 hours. Cost: a small GKE cluster costs about USD 0.30 an hour and the free trial credit covers it; this lab used the cluster for about 12 minutes. Delete the cluster at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud and kubectl); Cloud Shell in your browser has the same tools ready.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: a service mesh in plain English

In a microservices app, services call each other over the network all the time. Every team then needs the same things: encryption, retries, timeouts, canary releases and metrics. Without a mesh, each team writes that code again in each language. A service mesh moves it out of the app and into the network layer.

How Istio works
Istio's control plane gives rules and certificates to an Envoy sidecar next to every pod

Words you will use:

  • Sidecar: Istio adds a second container, the Envoy proxy, to every pod in the mesh. All traffic in and out of your app goes through it. Real-life example: a personal assistant who handles all your calls: answers, encrypts, tries again when the line is busy.
  • Control plane (istiod): gives every sidecar its rules and its certificate. The sidecars together are the data plane.
  • VirtualService: a routing rule: "for calls to web, send 90 percent here and 10 percent there".
  • DestinationRule: names the groups of pods behind a service (subsets, for example v1 and v2 by label).
  • Canary release: send a small share of real users to the new version first, watch it, then move more.
  • mTLS (mutual TLS): both sides show a certificate and the traffic is encrypted. PeerAuthentication STRICT = refuse anything that is not mTLS.
  • Retry: when a call fails with a 5xx error, the sidecar quietly tries again.

Before you start

  1. A Google Cloud account (the free trial works) and a project with billing linked.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console).
  3. Set your project and turn on the Kubernetes Engine API. Replace the text in quotes with your own project id:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com

Part B: the lab

Step 1: create a small GKE cluster and connect

Why: Istio runs inside Kubernetes. Two e2-standard-4 nodes, one zone, no GPU. We used this same cluster for four labs in a row, so it was created once:

gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --workload-pool=$PROJECT_ID.svc.id.goog
NAME         LOCATION    STATUS   CURRENT_NODE_COUNT  CURRENT_MASTER_VERSION
lab-cluster  us-east1-b  RUNNING  2                   1.36.4-gke.2377000
NAME          MACHINE_TYPE   DISK_SIZE_GB  INITIAL_NODE_COUNT
default-pool  e2-standard-4  50            2
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b 2>&1 | tail -1
kubectl get nodes
kubeconfig entry generated for lab-cluster.
NAME                                         STATUS   ROLES    AGE   VERSION
gke-lab-cluster-default-pool-27d9254d-1nv1   Ready    <none>   33m   v1.36.4-gke.2377000
gke-lab-cluster-default-pool-27d9254d-rl98   Ready    <none>   33m   v1.36.4-gke.2377000

Step 2: install istioctl and Istio

Why: istioctl is Istio's command line tool; the download script fetches the newest release. The minimal profile installs only the control plane (istiod), which is all you need for traffic rules and mTLS inside the cluster. (The demo and default profiles also add gateways for traffic from the internet.)

curl -sL https://istio.io/downloadIstio | sh - > /dev/null 2>&1
cd istio-*/ && sudo install -m 555 bin/istioctl /usr/local/bin/istioctl && cd ..
istioctl version --remote=false
# "minimal" = only the control plane (istiod): enough for traffic rules and mTLS inside the cluster
istioctl install --set profile=minimal -y 2>&1 | tail -3
kubectl -n istio-system get pods
client version: 1.32.0
โœ” CNI installed ๐Ÿชข
- Pruning removed resources
โœ” Installation completeNAME                      READY   STATUS    RESTARTS   AGE
istiod-7d4f59984d-hqvbp   1/1     Running   0          21s
Install Istio
Steps 1 and 2: istioctl 1.32.0, Istio installed, istiod Running

Step 3: deploy two versions, a flaky service and a client

Why: to see traffic splitting we need two versions of one service. web-v1 answers "web v1", web-v2 answers "web v2 (new)"; both carry the label app: web and one version label, and one Service web covers both. orders is go-httpbin, a test server whose page /status/200,503 answers 200 or 503 at random: our flaky service. client is a pod we send requests from. Save this as apps.yaml:

# Two versions of the same "web" service, a flaky "orders" service and a client pod to send requests from
apiVersion: apps/v1
kind: Deployment
metadata: {name: web-v1}
spec:
  selector: {matchLabels: {app: web, version: v1}}
  template:
    metadata: {labels: {app: web, version: v1}}
    spec:
      containers:
      - {name: web, image: "hashicorp/http-echo:1.0", args: ["-text=web v1", "-listen=:5678"], ports: [{containerPort: 5678}], resources: {requests: {cpu: 10m, memory: 16Mi}}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: web-v2}
spec:
  selector: {matchLabels: {app: web, version: v2}}
  template:
    metadata: {labels: {app: web, version: v2}}
    spec:
      containers:
      - {name: web, image: "hashicorp/http-echo:1.0", args: ["-text=web v2 (new)", "-listen=:5678"], ports: [{containerPort: 5678}], resources: {requests: {cpu: 10m, memory: 16Mi}}}
---
apiVersion: v1
kind: Service
metadata: {name: web}
spec:
  selector: {app: web}
  ports: [{name: http, port: 80, targetPort: 5678}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: orders}
spec:
  selector: {matchLabels: {app: orders}}
  template:
    metadata: {labels: {app: orders}}
    spec:
      containers:
      - {name: orders, image: "mccutchen/go-httpbin:v2.15.0", ports: [{containerPort: 8080}], resources: {requests: {cpu: 10m, memory: 16Mi}}}
---
apiVersion: v1
kind: Service
metadata: {name: orders}
spec:
  selector: {app: orders}
  ports: [{name: http, port: 80, targetPort: 8080}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: client}
spec:
  selector: {matchLabels: {app: client}}
  template:
    metadata: {labels: {app: client}}
    spec:
      containers:
      - {name: client, image: "curlimages/curl:8.10.1", command: ["sleep", "infinity"], resources: {requests: {cpu: 10m, memory: 16Mi}}}

The label istio-injection=enabled on the namespace tells Istio to add the sidecar to every new pod there.

kubectl create namespace mesh
kubectl label namespace mesh istio-injection=enabled
kubectl apply -n mesh -f apps.yaml
kubectl -n mesh rollout status deploy/web-v1 deploy/web-v2 deploy/orders deploy/client --timeout=180s > /dev/null
# READY 2/2 = your app container + the Envoy sidecar that Istio added
kubectl -n mesh get pods
namespace/mesh created
namespace/mesh labeled
deployment.apps/web-v1 created
deployment.apps/web-v2 created
service/web created
deployment.apps/orders created
service/orders created
deployment.apps/client created
NAME                      READY   STATUS    RESTARTS   AGE
client-bcfbbf867-lsl5g    2/2     Running   0          34s
orders-5d6947bfb4-w789r   2/2     Running   0          34s
web-v1-867d87f445-zf4fl   2/2     Running   0          35s
web-v2-7668659586-8677t   2/2     Running   0          35s

READY 2/2 on every pod: your container plus the Envoy sidecar.

Step 4: a canary release with a 90/10 traffic split

Why: first we send 100 requests with no Istio rule: Kubernetes spreads them over both versions, roughly half and half. That is a problem when v2 is new and maybe buggy. Then we add the rule. Save this as split-90-10.yaml:

# DestinationRule: name the two versions (subsets). VirtualService: send 90% to v1 and 10% to v2.
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: {name: web}
spec:
  host: web
  subsets:
  - {name: v1, labels: {version: v1}}
  - {name: v2, labels: {version: v2}}
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: {name: web}
spec:
  hosts: [web]
  http:
  - route:
    - {destination: {host: web, subset: v1}, weight: 90}
    - {destination: {host: web, subset: v2}, weight: 10}
# before any rule: Kubernetes spreads calls over all web pods (about 50/50)
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 100); do curl -s http://web; done' | sort | uniq -c
kubectl apply -n mesh -f split-90-10.yaml
sleep 5
echo "--- with the 90/10 rule ---"
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 100); do curl -s http://web; done' | sort | uniq -c
istioctl analyze -n mesh
     46 web v1
     54 web v2 (new)
destinationrule.networking.istio.io/web created
virtualservice.networking.istio.io/web created
--- with the 90/10 rule ---
     91 web v1
      9 web v2 (new)
โœ” No validation issues found when analyzing namespace: mesh.
Traffic split
Without a rule 46/54; with the 90/10 rule 91/9; istioctl analyze finds no issues

46/54 without the rule, 91/9 with it. The split is by chance per request, so it is close to 90/10, not exact. istioctl analyze checks your Istio config for mistakes (wrong host names, subsets that match no pods) and found none.

Step 5: move the canary to 50/50

Why: v2 looks fine, so we give it more traffic. This is just two numbers in the VirtualService, with no new pods and no restart. In real life you would watch the error rate and latency between each move.

# the canary looks fine: move to 50/50 by changing two numbers
kubectl -n mesh patch virtualservice web --type=json -p '[{"op":"replace","path":"/spec/http/0/route/0/weight","value":50},{"op":"replace","path":"/spec/http/0/route/1/weight","value":50}]'
sleep 5
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 100); do curl -s http://web; done' | sort | uniq -c
virtualservice.networking.istio.io/web patched
     49 web v1
     51 web v2 (new)
Shift to 50/50
Step 5: after the patch, 49 v1 and 51 v2

Break it and fix it: strict mTLS

We tell Istio to accept only mTLS traffic in the mesh namespace. Save this as mtls-strict.yaml:

# Only accept mTLS traffic (encrypted, with a workload certificate) in namespace "mesh"
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: mesh}
spec:
  mtls: {mode: STRICT}

Then a pod outside the mesh (namespace outside, no sidecar, so no certificate) calls web, and a pod inside calls it too:

kubectl apply -f mtls-strict.yaml
# Break it: a pod OUTSIDE the mesh (no sidecar, no certificate) calls web
kubectl create namespace outside
kubectl -n outside run outsider --image=curlimages/curl:8.10.1 --restart=Never -- sleep 3600
kubectl -n outside wait pod/outsider --for=condition=Ready --timeout=90s > /dev/null
sleep 5
kubectl -n outside exec outsider -- curl -s -m 5 http://web.mesh; echo "exit code from outside: $?"
# a pod inside the mesh still works (its sidecar does mTLS for it)
kubectl -n mesh exec deploy/client -- curl -s -m 5 http://web; echo "exit code from inside: $?"
peerauthentication.security.istio.io/default created
namespace/outside created
pod/outsider created
command terminated with exit code 56
exit code from outside: 56
web v1
exit code from inside: 0
mTLS blocks the outsider
From outside: curl exit code 56 (connection reset); from inside the mesh: web v1

The outsider's call fails with curl exit code 56 ("failure receiving data": the sidecar on the web side closed the plain-text connection). The client inside the mesh works, because its sidecar does mTLS for it. This is exactly what you want in production: a stolen or unknown workload cannot talk to your services. Fix: if the outsider is a real part of your system, bring it into the mesh.

# Fix it: bring the outside pod into the mesh (label the namespace, start the pod again)
kubectl label namespace outside istio-injection=enabled
kubectl -n outside delete pod outsider --wait=true > /dev/null
kubectl -n outside run outsider --image=curlimages/curl:8.10.1 --restart=Never -- sleep 3600
kubectl -n outside wait pod/outsider --for=condition=Ready --timeout=90s > /dev/null
kubectl -n outside get pod outsider
kubectl -n outside exec outsider -c outsider -- curl -s -m 5 http://web.mesh; echo "exit code: $?"
📝⭐ Recommended by CareerByteCode · Certification examGoogle Analytics Certification (GA4) PracticeTake the examTake the exam →
namespace/outside labeled
pod/outsider created
NAME       READY   STATUS    RESTARTS   AGE
outsider   2/2     Running   0          3s
web v2 (new)
exit code: 0
The outsider joins the mesh
The pod now shows 2/2 (with a sidecar) and gets an answer: exit code 0

The pod now has a sidecar (2/2), gets a certificate from istiod, and its call works.

Retries for a flaky service

Our orders service fails about half the time. Save this rule as orders-retries.yaml:

# Retry a failed call to "orders" up to 3 times when it answers 5xx
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: {name: orders}
spec:
  hosts: [orders]
  http:
  - route: [{destination: {host: orders}}]
    retries: {attempts: 3, perTryTimeout: 2s, retryOn: "5xx"}

We send 40 calls before and 40 after the rule:

# orders/status/200,503 answers 200 or 503 at random: a flaky service
echo "without retries:"
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code}\n" http://orders/status/200,503; done' | sort | uniq -c
kubectl apply -n mesh -f orders-retries.yaml
sleep 5
echo "with 3 retries on 5xx:"
kubectl -n mesh exec deploy/client -- sh -c 'for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code}\n" http://orders/status/200,503; done' | sort | uniq -c
without retries:
     19 200
     21 503
virtualservice.networking.istio.io/orders created
with 3 retries on 5xx:
     36 200
      4 503
Retries
Without retries 21 of 40 calls fail; with 3 retries only 4 fail

From 21 failures to 4, with no code change. Why not 0? Each try has about a 1 in 2 chance to fail, so all 4 tries (1 call plus 3 retries) fail about 1 time in 16: on 40 calls that is about 2 to 3. Retries hide short glitches, not a broken service, and they add load, so keep attempts small and only retry calls that are safe to repeat.

POC: prove it works

  • Traffic split: 91/9 with the 90/10 rule, 49/51 after the move to 50/50.
  • mTLS STRICT: a pod without a sidecar is refused (exit 56); the same pod inside the mesh works.
  • Retries: failed calls dropped from 21 of 40 to 4 of 40.
  • istioctl analyze: no validation issues.

Clean up (delete everything)

Why: remove the apps, Istio and its custom types, so the cluster is clean for the next lab or for deletion.

kubectl delete namespace mesh outside --wait=true
istioctl uninstall --purge -y 2>&1 | tail -1
kubectl delete namespace istio-system --wait=true
echo "namespaces left from this lab: $(kubectl get ns mesh outside istio-system --no-headers 2>/dev/null | wc -l)"
echo "Istio types left: $(kubectl get crd -o name | grep -c istio.io)"
namespace "mesh" deleted
namespace "outside" deleted
โœ” Uninstall completenamespace "istio-system" deleted
namespaces left from this lab: 0
Istio types left: 0

If you are not doing another Kubernetes lab now, delete the cluster (we did this after our fourth lab on it):

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet 2>&1 | tail -1
echo "clusters left: $(gcloud container clusters list --format="value(name)" | wc -l)"
echo "cluster disks left: $(gcloud compute disks list --filter="name~^gke-lab-cluster OR name~^pvc-" --format="value(name)" | wc -l)"
echo "load balancers left: $(gcloud compute forwarding-rules list --format="value(name)" | wc -l)"
Deleted [https://container.googleapis.com/v1/projects/your-project-id/zones/us-east1-b/clusters/lab-cluster].
clusters left: 0
cluster disks left: 0
load balancers left: 0
Clean up
0 lab namespaces, 0 Istio types, 0 clusters, 0 disks, 0 load balancers

FAQ

Do I need a service mesh? Not for 3 services. It pays off when many teams run many services and need the same security, traffic control and visibility everywhere. It also adds CPU and memory (one sidecar per pod) and something new to learn and run.

What about Istio ambient mode? Istio also has an "ambient" mode without sidecars (a per-node proxy plus optional waypoint proxies). This lab uses the classic sidecar mode, which is what most guides and interviews still describe.

Is GKE's own Cloud Service Mesh the same thing? Cloud Service Mesh is Google's managed offering built on Istio's APIs: the same VirtualService and DestinationRule ideas, with Google running the control plane. We tested open-source Istio only.

How do users from the internet reach a mesh service? Through an Istio ingress gateway (or the Kubernetes Gateway API). The minimal profile we used does not install one.

Why did the split show 91/9 and not exactly 90/10? Each request is routed by chance with those weights. Over thousands of requests it gets very close to 90/10.

Where do I see mesh metrics? Every sidecar publishes Prometheus metrics (requests, errors, latency per service). Add Prometheus and Grafana or Kiali to see them.

Troubleshooting

You seeWhyFix
Pods show 1/1, not 2/2The namespace has no istio-injection=enabled label, or the pods started before itLabel the namespace, then kubectl rollout restart deploy -n <ns>
The traffic split is ignoredNo DestinationRule subsets, or labels do not matchCheck version labels; run istioctl analyze
curl exit code 56 between servicesSTRICT mTLS and the caller has no sidecarBring the caller into the mesh, or use PERMISSIVE while migrating
Retries do not helpThe service fails every time, or the error is not 5xxFix the service; check retryOn
istioctl install hangsNodes too smallUse e2-standard-4 nodes or the minimal profile
Exec into a pod asks "Defaulted container"The pod has two containers nowAdd -c <container name>

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Networking Interview Questions for Cloud Engineers, Proved in Labs⏱ 11 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →