KServe Hands-On: Serverless Model Serving on Kubernetes
What you build: KServe on GKE serving a scikit-learn model and a small language model on CPU, a broken request, and autoscaling under load. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
You trained a model. Now people want to call it: a web address, a JSON request, an answer in milliseconds, more copies when traffic grows. KServe is the standard open-source way to do this on Kubernetes: you describe the model in one small YAML object, and KServe downloads it, starts a server, gives it an address and scales it. In this lab you serve a scikit-learn model and a small language model on GKE (CPU only), send a broken request on purpose, and watch the autoscaler add copies under load.
Who it is for: ML engineers, MLOps beginners and Kubernetes users who want to see model serving work end to end. Time: about 1 hour. Cost: a small GKE cluster for about 15 minutes, covered by the free trial credit. Delete it at the end.
Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl and helm); Cloud Shell in your browser has the same tools.
Part A: model serving in plain English
A model server is a small web service that loads a model file and answers prediction requests. Writing one by hand for every model gets old fast. KServe gives you ready-made servers (serving runtimes) for the common model formats, and one object type, the InferenceService, to run them.

Words you will use:
- InferenceService: the YAML object that says "serve this model, in this format, with these resources". KServe turns it into a Deployment, a Service and an autoscaler.
- Serving runtime: the server program for one model format:
sklearn,xgboost,huggingface,tensorflowand more. KServe chooses it frommodelFormat. - storageUri: where the model file lives, for example a Cloud Storage path (
gs://...). KServe downloads it before the server starts. - Predictor: the pods that actually answer requests.
- RawDeployment mode: KServe uses plain Kubernetes Deployments and a Horizontal Pod Autoscaler (HPA). The other mode uses Knative and can scale to zero; RawDeployment needs fewer extra parts, so we use it.
- V1 inference protocol:
POST /v1/models/<name>:predictwith{"instances": [...]}, answered with{"predictions": [...]}. - Real-life example: an InferenceService is an order at a print shop. You hand in the file and say "50 copies, colour". You do not choose the printer or load the paper yourself.
Before you start
- A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
- One small GKE cluster. We created it once for this batch of Kubernetes labs:
gcloud services enable container.googleapis.com
gcloud container clusters create lab-cluster \
--zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
--disk-type=pd-standard --disk-size=50 --release-channel=regular \
--no-enable-autoupgrade --enable-dataplane-v2Run every step below in the same Cloud Shell folder (Step 2 and Step 5 save YAML files there).
Part B: the lab
Step 1: install cert-manager and KServe
Why: KServe checks every InferenceService with webhooks, and webhooks need TLS certificates; cert-manager creates them. Then we install KServe v0.14.1 and its cluster resources, which include the serving runtimes.
# Install cert-manager (KServe needs it for its webhooks), then KServe in RawDeployment mode (plain Deployments + HPA, no Knative)
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml > /dev/null && kubectl -n cert-manager rollout status deploy/cert-manager-webhook --timeout=180s
kubectl apply --server-side -f https://github.com/kserve/kserve/releases/download/v0.14.1/kserve.yaml > /dev/null 2>&1; kubectl -n kserve rollout status deploy/kserve-controller-manager --timeout=300s
sleep 20; kubectl apply --server-side -f https://github.com/kserve/kserve/releases/download/v0.14.1/kserve-cluster-resources.yaml > /dev/null 2>&1 && kubectl get clusterservingruntimes --no-headers | awk '{print $1}' | tr '\n' ' '; echoWaiting for deployment "cert-manager-webhook" rollout to finish: 0 of 1 updated replicas are available...
deployment "cert-manager-webhook" successfully rolled out
Waiting for deployment "kserve-controller-manager" rollout to finish: 0 of 1 updated replicas are available...
deployment "kserve-controller-manager" successfully rolled out
kserve-huggingfaceserver kserve-lgbserver kserve-mlserver kserve-paddleserver kserve-pmmlserver kserve-sklearnserver kserve-tensorflow-serving kserve-torchserve kserve-tritonserver kserve-xgbserverThe last line lists the runtimes KServe now knows: scikit-learn, Hugging Face, XGBoost, TensorFlow, Triton and more.
Step 2: serve a scikit-learn model
Why: this is the whole "deployment" of a model: one object. The model is a public example model from the KServe project for the iris flower data set (four measurements per flower, three species), kept in a public Cloud Storage bucket. The comment in the file says "Google-hosted": that only means it sits in Cloud Storage. The annotations ask for RawDeployment mode and an HPA that adds copies when CPU passes 30 percent of the request, up to 3 copies. Save this as sklearn.yaml:
# A scikit-learn model (Google-hosted public example: iris flowers) served by KServe, scaled by an HPA on CPU
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: sklearn-iris
namespace: models
annotations:
serving.kserve.io/deploymentMode: RawDeployment
serving.kserve.io/autoscalerClass: hpa
serving.kserve.io/metric: cpu
serving.kserve.io/targetUtilizationPercentage: "30"
spec:
predictor:
minReplicas: 1
maxReplicas: 3
model:
modelFormat: {name: sklearn}
storageUri: gs://kfserving-examples/models/sklearn/1.0/model
resources:
requests: {cpu: 100m, memory: 256Mi}
limits: {cpu: 500m, memory: 512Mi}# Serve a scikit-learn model: one InferenceService object. KServe downloads the model and starts a predictor
kubectl create namespace models > /dev/null && kubectl apply -f sklearn.yaml
for i in $(seq 1 60); do R=$(kubectl -n models get isvc sklearn-iris -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'); [ "$R" = True ] && break; sleep 5; done; echo "sklearn-iris ready: $R"
kubectl -n models get isvc sklearn-iris -o custom-columns=NAME:.metadata.name,URL:.status.url,READY:.status.conditions[-1].statusinferenceservice.serving.kserve.io/sklearn-iris created
sklearn-iris ready: True
NAME URL READY
sklearn-iris http://sklearn-iris-models.example.com TrueREADY True means KServe downloaded the model, started the scikit-learn server and it passed its health check. The URL uses example.com because we did not set a domain; inside the cluster the predictor is reachable as the Service sklearn-iris-predictor.
Step 3: ask the model
Why: a model is only useful when something can call it. We open a tunnel to the predictor with kubectl port-forward and send two flowers, four numbers each:
# Ask the model: which iris species are these two flowers? (V1 protocol: POST /v1/models/<name>:predict)
kubectl -n models port-forward svc/sklearn-iris-predictor 8081:80 > /dev/null 2>&1 & sleep 4
curl -s -H "Content-Type: application/json" -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}' localhost:8081/v1/models/sklearn-iris:predict; echo
curl -s -o /dev/null -w "HTTP %{http_code} for a model that does not exist\n" -H "Content-Type: application/json" -d '{"instances": [[1, 2, 3, 4]]}' localhost:8081/v1/models/no-such-model:predict
kill %1{"predictions":[1,1]}
HTTP 404 for a model that does not existBoth flowers are class 1 (versicolor, the middle species). A request for a model name that does not exist gets HTTP 404, which is what a client should see.

Break it: the wrong input shape
The model was trained on 4 numbers per flower. We send 3, a very common bug when a feature is added or removed in one place but not the other:
# Break it: send the wrong input shape (3 numbers instead of 4 per flower)
kubectl -n models port-forward svc/sklearn-iris-predictor 8081:80 > /dev/null 2>&1 & sleep 4
curl -s -w "\nHTTP %{http_code}\n" -H "Content-Type: application/json" -d '{"instances": [[6.8, 2.8, 4.8]]}' localhost:8081/v1/models/sklearn-iris:predict | cut -c1-200
kill %1{"error":"X has 3 features, but SVC is expecting 4 features as input."}
HTTP 500The server answers HTTP 500 with the model's own message: "X has 3 features, but SVC is expecting 4". The fix is in the client: send the same features, in the same order, that the model was trained with. Teams keep that list (the model signature) next to the model and check it in tests.
Step 4: autoscaling under load
Why: traffic is never flat. We start a load generator (fortio) inside the cluster with 16 parallel connections for 2 minutes, and watch the HPA:
# Autoscaling: load the predictor for 2 minutes and watch the HPA add pods (1 to 3) for 90 s
kubectl -n models get hpa -o custom-columns=NAME:.metadata.name,MIN:.spec.minReplicas,MAX:.spec.maxReplicas,TARGET:.spec.metrics[0].resource.target.averageUtilization
kubectl -n models run loadgen --image=fortio/fortio:1.60.3 --restart=Never -- load -qps 0 -c 16 -t 120s -quiet -payload '{"instances": [[6.8, 2.8, 4.8, 1.4]]}' -content-type application/json http://sklearn-iris-predictor/v1/models/sklearn-iris:predict > /dev/null
for t in 30 60 90; do sleep 30; echo "after ${t}s: $(kubectl -n models get deploy sklearn-iris-predictor -o jsonpath='{.status.replicas}') predictor pods, cpu now $(kubectl -n models get hpa -o jsonpath='{.items[0].status.currentMetrics[0].resource.current.averageUtilization}')%"; done
kubectl -n models wait pod/loadgen --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s > /dev/null; kubectl -n models logs loadgen | grep -E "All done|Code 200|^# target 99%" | head -3; kubectl -n models delete pod loadgen > /dev/nullNAME MIN MAX TARGET
sklearn-iris-predictor 1 3 30
after 30s: 3 predictor pods, cpu now 75%
after 60s: 3 predictor pods, cpu now 500%
after 90s: 3 predictor pods, cpu now 500%
# target 99% 0.0941197
Code 200 : 33053 (100.0 %)
All done 33053 calls (plus 16 warmup) 58.103 ms avg, 275.3 qpsWithin 30 seconds the HPA went from 1 to 3 predictor pods (its maximum). The CPU number is the share of the pod's CPU request (100m): 500 percent means each pod used its full limit of 500m. Over 33,000 predictions came back, all with HTTP 200, at about 275 a second, and 99 percent of them in under 0.1 seconds.
🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโน10,620See the training →
Step 5: a small language model on CPU
Why: the same InferenceService works for Hugging Face models. We serve DistilBERT fine-tuned for sentiment (about 260 MB), with KServe's Hugging Face runtime. No storageUri: the runtime downloads the model by its Hugging Face name. Save this as sentiment.yaml:
# A small language model (DistilBERT sentiment, about 260 MB) served by KServe's Hugging Face runtime on CPU
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: sentiment
namespace: models
annotations:
serving.kserve.io/deploymentMode: RawDeployment
spec:
predictor:
model:
modelFormat: {name: huggingface}
args:
- --model_name=sentiment
- --model_id=distilbert/distilbert-base-uncased-finetuned-sst-2-english
- --task=sequence_classification
resources:
requests: {cpu: "1", memory: 2Gi}
limits: {cpu: "2", memory: 3Gi}# A small language model on CPU: DistilBERT sentiment with KServe's Hugging Face runtime (the image is large: a few minutes)
kubectl apply -f sentiment.yaml
for i in $(seq 1 90); do R=$(kubectl -n models get isvc sentiment -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'); [ "$R" = True ] && break; sleep 10; done; echo "sentiment ready: $R"
kubectl -n models port-forward svc/sentiment-predictor 8082:80 > /dev/null 2>&1 & sleep 4
curl -s -H "Content-Type: application/json" -d '{"instances": ["This lab finally made model serving click for me.", "The deployment failed again and nobody knows why."]}' localhost:8082/v1/models/sentiment:predict; echo
kill %1inferenceservice.serving.kserve.io/sentiment created
sentiment ready: True
{"predictions":[1,0]}[1, 0] means the first sentence is positive and the second negative. The image is large, so this one took about 5 minutes to become ready. That start-up time matters when you plan autoscaling for big models.
POC: prove it works
- One InferenceService became a ready model server with an address (Step 2).
- It answered predictions, returned 404 for an unknown model and a clear 500 for a wrong input shape (Step 3 and Break it).
- Under load the HPA scaled from 1 to 3 pods and every one of 33,053 requests succeeded (Step 4).
- A Hugging Face language model ran on CPU with the same kind of object (Step 5).
Clean up (delete everything)
Why: model servers hold memory and CPU while they run. We delete the models, KServe and cert-manager.
# Clean up: the models, KServe and cert-manager
kubectl delete namespace models --wait=true 2>&1 | tail -1
kubectl delete -f https://github.com/kserve/kserve/releases/download/v0.14.1/kserve-cluster-resources.yaml --ignore-not-found > /dev/null 2>&1
kubectl delete -f https://github.com/kserve/kserve/releases/download/v0.14.1/kserve.yaml --ignore-not-found > /dev/null 2>&1; echo "kserve removed"
kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml --ignore-not-found > /dev/null 2>&1; echo "cert-manager removed"
echo "namespaces left: $(kubectl get ns --no-headers | grep -cE '^(models|kserve|cert-manager) ')"namespace "models" deleted
kserve removed
cert-manager removed
namespaces left: 0When you have finished all your Kubernetes labs, delete the cluster:
gcloud container clusters delete lab-cluster --zone=us-east1-b --quietFAQ
RawDeployment or Serverless mode? Serverless (with Knative) can scale to zero and wake up on the first request, which saves resources for models that are rarely used. RawDeployment is simpler and has no cold start. We tested RawDeployment only.
Do I need GPUs for LLMs? For small models like DistilBERT, no. For large language models, yes in practice; KServe then uses nvidia.com/gpu in resources. This lab is CPU only.
Where should the model file live? In Cloud Storage or another object store, with a version in the path (gs://my-models/churn/v3/). Then a rollback is a one-line change of storageUri.
How do I roll out a new model safely? KServe supports canary rollouts with canaryTrafficPercent in Serverless mode. We did not test it here.
Can I use my own server code? Yes: KServe accepts a custom container under predictor.containers, as long as it speaks the inference protocol.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
READY stays empty for minutes | The runtime image or the model is still downloading | kubectl -n models describe pod shows the progress; wait |
no matches for kind "InferenceService" | KServe is not installed yet | Install cert-manager, then KServe, then the cluster resources |
| Webhook errors when creating the service | cert-manager was not ready yet | Wait for cert-manager-webhook to roll out before KServe |
| HTTP 500 "expecting 4 features" | The client sends a different input shape | Send the features the model was trained with |
HPA shows <unknown> CPU | The pods have no CPU request | Set resources.requests.cpu in the predictor |
| The language model pod restarts | Not enough memory | Give it at least 2 GiB, as in sentiment.yaml |
What next
- Run an LLM with a different tool: Run Your Own LLM on Kubernetes with Ollama.
- Get a model ready for serving: MLOps for Beginners: From Notebook to Production.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamRay on Kubernetes with KubeRay: Distributed Python for AI
What you build: a Ray cluster on GKE with KubeRay, parallel Python tasks, a stuck oversized task and a self-cleaning RayJob. Every step tested.
#ai
CareerByteCode TeamAI Infrastructure Engineer Interview Questions with Labs
What you build: 5 AI infrastructure interview answers proved on a CPU VM: model memory, batching, quantization, CPU throttling and model storage speed. Every step tested.
#ai-infrastructure
CareerByteCode TeamRun Your Own LLM on Kubernetes with Ollama (CPU Only)
What you build: Ollama on a CPU-only GKE node with Llama 3.2 1B and Qwen 2.5 0.5B on a persistent disk, real answers and speed in tokens per second, in your own Google Cloud account. Every step tested
#chaos
CareerByteCode TeamChaos Engineering on Kubernetes with Chaos Mesh
What you build: Chaos Mesh on GKE killing pods and slowing the network, a weak setting found and fixed with real success rates. Every step tested.
💬 Comments (0)
Top · NewestSign in to comment