Run Your Own LLM on Kubernetes with Ollama (CPU Only)
What you build: Ollama on a CPU-only GKE node with Llama 3.2 1B and Qwen 2.5 0.5B on a persistent disk, real answers and speed in tokens per second, in your own Google Cloud account. Every step tested

✍️ Translate this blog into Hindi, Tamil or Telugu
In this lab you run your own large language model (LLM) on Kubernetes, with no GPU and no API key: Ollama in one pod on a normal CPU node of Google Kubernetes Engine (GKE), two small open models (Llama 3.2 1B and Qwen 2.5 0.5B) on a persistent disk, and real questions answered through Ollama's HTTP API. You measure the speed in tokens per second, break the pod by giving it too little memory, and fix it.
Who it is for: beginners who know basic Kubernetes and want to understand what "self-hosting an LLM" really means, before paying for GPUs. Time: about 1 to 1.5 hours. Cost: a small GKE cluster costs about USD 0.30 an hour and the free trial credit covers it; this lab used the cluster for about 15 minutes. Delete the cluster and its disks at the end.
Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl, curl and jq); Cloud Shell in your browser has the same tools ready.
Part A: running an LLM yourself, in plain English
An LLM is a program that has learned from a huge amount of text and predicts the next word, again and again, to write an answer. The learned knowledge is stored as parameters (numbers): "1B" means 1 billion of them. Bigger models know more and write better, but need more memory and are slower.

Words you will use:
- Ollama: an open-source program that downloads open models and serves them over a simple HTTP API on port 11434. Real-life example: a jukebox for AI models: you pick a model by name and it plays (answers).
- Model file: the parameters on disk.
llama3.2:1bis 1.3 GB,qwen2.5:0.5bis 397 MB. They are quantized: the numbers are stored with fewer bits so they fit in less memory with a small loss of quality. - Token: a piece of a word. Speed is measured in tokens per second; a person reads about 5 to 8 tokens a second.
- CPU versus GPU: a GPU does the same maths thousands of times in parallel, so it is many times faster for LLMs. A small model on a 4-vCPU node is slower but works, and costs far less.
- PersistentVolumeClaim (PVC): a request for a disk that survives pod restarts. We keep the models on it so a restarted pod does not download them again.
- Requests and limits: what the pod is promised (requests) and the most it may use (limits). Over the memory limit, Kubernetes kills the container: OOMKilled (out of memory).
Before you start
- A Google Cloud account (the free trial works) and a project with billing linked.
- Open Cloud Shell (the
>_icon at the top right of the Google Cloud console). - Set your project and turn on the Kubernetes Engine API. Replace the text in quotes with your own project id:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.comPart B: the lab
Step 1: create a small GKE cluster (CPU only) and connect
Why: Ollama runs in a pod. Two e2-standard-4 nodes (4 vCPU, 16 GB memory each), one zone, no GPU. We used this same cluster for four labs in a row, so it was created once:
gcloud container clusters create lab-cluster \
--zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
--disk-type=pd-standard --disk-size=50 --release-channel=regular \
--no-enable-autoupgrade --workload-pool=$PROJECT_ID.svc.id.googNAME LOCATION STATUS CURRENT_NODE_COUNT CURRENT_MASTER_VERSION
lab-cluster us-east1-b RUNNING 2 1.36.4-gke.2377000
NAME MACHINE_TYPE DISK_SIZE_GB INITIAL_NODE_COUNT
default-pool e2-standard-4 50 2gcloud container clusters get-credentials lab-cluster --zone=us-east1-b 2>&1 | tail -1
kubectl get nodeskubeconfig entry generated for lab-cluster.
NAME STATUS ROLES AGE VERSION
gke-lab-cluster-default-pool-27d9254d-1nv1 Ready <none> 39m v1.36.4-gke.2377000
gke-lab-cluster-default-pool-27d9254d-rl98 Ready <none> 39m v1.36.4-gke.2377000Step 2: deploy Ollama with a disk for the models
Why: one Deployment with one pod, a 10 GB PVC for the models, and a Service. The pod asks for 3 CPUs and 4 GiB memory and may use up to 6 GiB. strategy: Recreate stops the old pod before starting a new one, because a normal disk can be attached to only one pod at a time. OLLAMA_KEEP_ALIVE=30m keeps a loaded model in memory for 30 minutes so repeat questions are fast. Save this as ollama.yaml:
# Ollama on CPU: one pod, a 10 GB disk for the models, and a service on port 11434
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ollama-models}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 10Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ollama}
spec:
replicas: 1
strategy: {type: Recreate}
selector: {matchLabels: {app: ollama}}
template:
metadata: {labels: {app: ollama}}
spec:
containers:
- name: ollama
image: ollama/ollama:latest
ports: [{containerPort: 11434}]
env: [{name: OLLAMA_KEEP_ALIVE, value: "30m"}]
resources:
requests: {cpu: "3", memory: 4Gi}
limits: {memory: 6Gi}
readinessProbe: {httpGet: {path: /api/version, port: 11434}, periodSeconds: 5}
volumeMounts: [{name: models, mountPath: /root/.ollama}]
volumes:
- {name: models, persistentVolumeClaim: {claimName: ollama-models}}
---
apiVersion: v1
kind: Service
metadata: {name: ollama}
spec:
selector: {app: ollama}
ports: [{port: 11434, targetPort: 11434}]kubectl create namespace llm
kubectl apply -n llm -f ollama.yaml
kubectl -n llm rollout status deploy/ollama --timeout=300s
kubectl -n llm get pods,pvc
# open the Ollama API on localhost:11434 (keeps running in the background)
nohup kubectl -n llm port-forward svc/ollama 11434:11434 > pf-ollama.log 2>&1 &
sleep 4; curl -s http://localhost:11434/api/version; echonamespace/llm created
persistentvolumeclaim/ollama-models created
deployment.apps/ollama created
service/ollama created
Waiting for deployment "ollama" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama" successfully rolled out
NAME READY STATUS RESTARTS AGE
pod/ollama-85479f4c87-sqfhv 1/1 Running 0 3m2s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
persistentvolumeclaim/ollama-models Bound pvc-500eeba1-c6d1-4036-badc-275ceca93a31 10Gi RWO standard-rwo <unset> 3m2s
{"version":"0.40.16"}
The first start took about 3 minutes: GKE created the disk and the node downloaded the Ollama image (about 1 GB). {"version":"0.40.16"} is Ollama answering on localhost:11434 through kubectl port-forward.
Step 3: download two small open models
Why: ollama pull downloads a model into the pod's disk. We take two: Llama 3.2 1B (Meta) and Qwen 2.5 0.5B (Alibaba), both made to run on small machines. Check each model's licence before you use it in a product.
# download two small open models into the pod's disk (1B and 0.5B parameters)
kubectl -n llm exec deploy/ollama -- ollama pull llama3.2:1b 2>&1 | tail -1
kubectl -n llm exec deploy/ollama -- ollama pull qwen2.5:0.5b 2>&1 | tail -1
kubectl -n llm exec deploy/ollama -- ollama listsuccess
success
NAME ID SIZE MODIFIED
qwen2.5:0.5b a8b0c5157701 397 MB Less than a second ago
llama3.2:1b baf6a787fdff 1.3 GB 5 seconds agoStep 4: ask the model a question
Why: Ollama's /api/generate takes a JSON request: the model name, the prompt, stream: false (one answer at the end instead of word by word), a low temperature for steady answers and num_predict to cap the answer at 120 tokens. The answer JSON also has timing fields, which we turn into tokens per second with jq.
cat > ask.json <<'JSON'
{"model": "llama3.2:1b", "stream": false,
"prompt": "Explain what a Kubernetes pod is to a beginner in 2 short sentences.",
"options": {"temperature": 0.2, "num_predict": 120}}
JSON
curl -s http://localhost:11434/api/generate -d @ask.json > answer.json
jq -r '.response' answer.json
jq -r '"tokens out: \(.eval_count), speed: \((.eval_count / (.eval_duration / 1e9)) * 10 | round / 10) tokens per second, total time: \((.total_duration / 1e9) * 10 | round / 10) s (load \((.load_duration / 1e9) * 10 | round / 10) s)"' answer.jsonA Kubernetes pod is a basic execution unit in Kubernetes that represents a single instance of a container, consisting of one or more containers that run together. Pods are the building blocks of Kubernetes deployments and can be scaled up or down based on demand, allowing for efficient resource utilization and high availability.
tokens out: 58, speed: 10.2 tokens per second, total time: 9.7 s (load 2.9 s)
A correct, readable answer from a model running on your own CPU node. 10.2 tokens a second is about reading speed. load 2.9 s is the time to load the model from disk into memory the first time.
Step 5: measure the speed of both models
Why: model size is the main speed lever on CPU. We ask both models the same question twice.
# same question to both models, 2 rounds each: compare speed on CPU
for m in llama3.2:1b qwen2.5:0.5b; do
for r in 1 2; do
jq -n --arg m "$m" '{model: $m, stream: false, prompt: "List 3 benefits of containers in one line each.", options: {temperature: 0.2, num_predict: 80}}' > q.json
curl -s http://localhost:11434/api/generate -d @q.json | jq -r --arg m "$m" --arg r "$r" '"\($m) round \($r): \(.eval_count) tokens, \((.eval_count / (.eval_duration / 1e9)) * 10 | round / 10) tokens/s, \((.total_duration / 1e9) * 10 | round / 10) s total"'
done
done
# how much CPU and memory the pod used (metrics from the last minute)
kubectl -n llm top podllama3.2:1b round 1: 55 tokens, 10.3 tokens/s, 5.8 s total
llama3.2:1b round 2: 61 tokens, 10.3 tokens/s, 6.1 s total
qwen2.5:0.5b round 1: 80 tokens, 23.7 tokens/s, 5.5 s total
qwen2.5:0.5b round 2: 80 tokens, 23.7 tokens/s, 3.4 s total
NAME CPU(cores) MEMORY(bytes)
ollama-85479f4c87-sqfhv 368m 277Mi
| Model | Size on disk | Speed on 3 vCPU (our test) | Good for |
|---|---|---|---|
| llama3.2:1b | 1.3 GB | about 10 tokens per second | short answers, summaries, classification |
| qwen2.5:0.5b | 397 MB | about 24 tokens per second | very simple tasks, fast tests |
The second Qwen round was faster in total (3.4 s against 5.5 s) because the model was already in memory. kubectl top showed the pod using little CPU at that moment, because metrics are an average over the last minute and the pod is idle between questions. Watch the cost: the node costs the same whether the model is busy or idle, so a self-hosted model is cheap only when it is used a lot; for a few questions a day, a pay-per-token API is cheaper. To serve more users, add replicas, which each need their own CPU and memory (and their own copy of the models, because a normal disk attaches to one pod).
Break it and fix it: too little memory
We set the memory limit to 512 MiB, far less than the 1B model needs, and ask the same question again.
# Break it: squeeze the memory limit to 512 MiB (the 1B model needs more than that)
kubectl -n llm set resources deploy/ollama --limits=memory=512Mi --requests=memory=256Mi
kubectl -n llm rollout status deploy/ollama --timeout=180s
pkill -f "[p]ort-forward svc/ollama"; nohup kubectl -n llm port-forward svc/ollama 11434:11434 > pf-ollama.log 2>&1 &
sleep 4
curl -s -m 120 http://localhost:11434/api/generate -d @ask.json | head -c 300; echo
sleep 5
kubectl -n llm get pods
kubectl -n llm get pod -l app=ollama -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'; echo🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโน10,620See the training →deployment.apps/ollama resource requirements updated
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama" successfully rolled out
NAME READY STATUS RESTARTS AGE
ollama-7bbdc9cd9f-plqx9 1/1 Running 1 (7s ago) 19s
OOMKilled
The answer is empty (the connection dropped), the pod shows RESTARTS 1 and the last termination reason is OOMKilled: Ollama tried to load the model into memory, went over 512 MiB, and the kernel killed it. Fix: give it enough memory again.
# Fix it: give the pod enough memory again
kubectl -n llm set resources deploy/ollama --limits=memory=6Gi --requests=memory=4Gi
kubectl -n llm rollout status deploy/ollama --timeout=180s
pkill -f "[p]ort-forward svc/ollama"; nohup kubectl -n llm port-forward svc/ollama 11434:11434 > pf-ollama.log 2>&1 &
sleep 4
curl -s http://localhost:11434/api/generate -d @ask.json | jq -r '.response'
kubectl -n llm get pods
# the models were still on the disk (PersistentVolumeClaim), so nothing had to be downloaded again
kubectl -n llm exec deploy/ollama -- ollama listdeployment.apps/ollama resource requirements updated
Waiting for deployment spec update to be observed...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama" successfully rolled out
A Kubernetes pod is a basic execution unit in a Kubernetes cluster, consisting of one or more containers that run together on a single host, providing a single IP address and network identity. Pods are the building blocks of Kubernetes applications, and they can be scaled up or down based on demand, allowing for efficient resource utilization and high availability.
NAME READY STATUS RESTARTS AGE
ollama-85479f4c87-22l98 1/1 Running 0 2m29s
NAME ID SIZE MODIFIED
qwen2.5:0.5b a8b0c5157701 397 MB 4 minutes ago
llama3.2:1b baf6a787fdff 1.3 GB 4 minutes ago
The models were still on the PVC, so nothing was downloaded again. Rule of thumb for CPU serving: memory limit of at least 2 to 3 times the model file size.
POC: prove it works
- Ollama runs as a normal Kubernetes Deployment, with its models on a persistent disk.
- Llama 3.2 1B answered a real question at about 10 tokens per second on CPU; Qwen 2.5 0.5B at about 24.
- Too little memory gives
OOMKilled; the right limits fix it, with no new download. - No GPU, no API key, no data leaves your cluster.
Clean up (delete everything)
Why: the PVC is a real Compute Engine disk that adds to your bill even when no pod uses it. Deleting the namespace deletes the PVC, and GKE then deletes the disk.
pkill -f "[p]ort-forward svc/ollama"
kubectl delete namespace llm --wait=true
# the model disk was a Compute Engine persistent disk; deleting the claim deletes it
sleep 20
echo "namespaces left from this lab: $(kubectl get ns llm --no-headers 2>/dev/null | wc -l)"
echo "model disks left: $(gcloud compute disks list --filter="name~^pvc-" --format="value(name)" | wc -l)"namespace "llm" deleted
namespaces left from this lab: 0
model disks left: 0If you are not doing another Kubernetes lab now, delete the cluster (we did this right after this lab, our fourth on it):
gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet 2>&1 | tail -1
echo "clusters left: $(gcloud container clusters list --format="value(name)" | wc -l)"
echo "cluster disks left: $(gcloud compute disks list --filter="name~^gke-lab-cluster OR name~^pvc-" --format="value(name)" | wc -l)"
echo "load balancers left: $(gcloud compute forwarding-rules list --format="value(name)" | wc -l)"Deleted [https://container.googleapis.com/v1/projects/your-project-id/zones/us-east1-b/clusters/lab-cluster].
clusters left: 0
cluster disks left: 0
load balancers left: 0
FAQ
Can I run bigger models like 7B or 8B on CPU? They run, but slowly (often 2 to 5 tokens per second on 4 vCPUs) and need 8 GB or more memory. For those, use a GPU node pool. GPU labs are a separate post.
Is a self-hosted model private? The prompts and answers stay in your cluster. That is a common reason companies self-host. You still own security: keep the service internal, as here (ClusterIP and port-forward only).
Ollama or vLLM? Ollama is the easiest way to start and runs well on CPU. vLLM is built for high-throughput GPU serving in production.
How do I call it from another app in the cluster? Use the service address http://ollama.llm:11434/api/generate (service ollama, namespace llm).
Does it have an OpenAI-style API? Yes, Ollama also answers on /v1/chat/completions. We tested only /api/generate in this lab.
Why is the first answer slower? The model is loaded from disk into memory first (load_duration). With OLLAMA_KEEP_ALIVE it stays loaded.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
Pod Pending with "Insufficient cpu" or "memory" | The node is too small for the requests | Use e2-standard-4 nodes or lower the requests |
Pod restarts, reason OOMKilled | Memory limit below what the model needs | Raise the limit (2 to 3 times the model size) |
Multi-Attach error for the volume | Two pods want the same ReadWriteOnce disk | Use strategy: Recreate and one replica per disk |
connection refused on localhost:11434 | The port-forward stopped (for example after a pod restart) | Start kubectl -n llm port-forward svc/ollama 11434:11434 again |
model "x" not found | The model was not pulled | kubectl -n llm exec deploy/ollama -- ollama pull x |
| Very slow answers | Big model or few CPUs | Use a smaller model or give the pod more CPUs |
What next
- New to Kubernetes? Start with Zero to Kubernetes in 2.5 Hours.
- Compare with a managed model: Gemini API for Beginners: Your First AI App on Google Cloud.
- Expose your model to AI agents: Model Context Protocol (MCP) Server on Cloud Run.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamRay on Kubernetes with KubeRay: Distributed Python for AI
What you build: a Ray cluster on GKE with KubeRay, parallel Python tasks, a stuck oversized task and a self-cleaning RayJob. Every step tested.
#beginners
CareerByteCode TeamKubernetes Roadmap: From Pods to Platform Engineering
What you build: an 8-level Kubernetes roadmap from pods to platform engineering, with a real script that checks every level on GKE (pod, service, config, probes, HPA, RBAC, Helm, as code).
#ai-infrastructure
CareerByteCode TeamKServe Hands-On: Serverless Model Serving on Kubernetes
What you build: KServe on GKE serving a scikit-learn model and a small language model on CPU, a broken request, and autoscaling under load. Every step tested.
#ai
CareerByteCode TeamAI Infrastructure Engineer Interview Questions with Labs
What you build: 5 AI infrastructure interview answers proved on a CPU VM: model memory, batching, quantization, CPU throttling and model storage speed. Every step tested.
💬 Comments (0)
Top · NewestSign in to comment