CareerByteCode
ByteBlogby CareerByteCode

Run Your Own LLM on Kubernetes with Ollama (CPU Only)

What you build: Ollama on a CPU-only GKE node with Llama 3.2 1B and Qwen 2.5 0.5B on a persistent disk, real answers and speed in tokens per second, in your own Google Cloud account. Every step tested

Run Your Own LLM on Kubernetes with Ollama (CPU Only)

✍️ Translate this blog into Hindi, Tamil or Telugu

In this lab you run your own large language model (LLM) on Kubernetes, with no GPU and no API key: Ollama in one pod on a normal CPU node of Google Kubernetes Engine (GKE), two small open models (Llama 3.2 1B and Qwen 2.5 0.5B) on a persistent disk, and real questions answered through Ollama's HTTP API. You measure the speed in tokens per second, break the pod by giving it too little memory, and fix it.

Who it is for: beginners who know basic Kubernetes and want to understand what "self-hosting an LLM" really means, before paying for GPUs. Time: about 1 to 1.5 hours. Cost: a small GKE cluster costs about USD 0.30 an hour and the free trial credit covers it; this lab used the cluster for about 15 minutes. Delete the cluster and its disks at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl, curl and jq); Cloud Shell in your browser has the same tools ready.

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →

Part A: running an LLM yourself, in plain English

An LLM is a program that has learned from a huge amount of text and predicts the next word, again and again, to write an answer. The learned knowledge is stored as parameters (numbers): "1B" means 1 billion of them. Bigger models know more and write better, but need more memory and are slower.

How Ollama on Kubernetes works
Your request goes to the Ollama service, the pod loads the model from its disk into memory and answers on CPU

Words you will use:

  • Ollama: an open-source program that downloads open models and serves them over a simple HTTP API on port 11434. Real-life example: a jukebox for AI models: you pick a model by name and it plays (answers).
  • Model file: the parameters on disk. llama3.2:1b is 1.3 GB, qwen2.5:0.5b is 397 MB. They are quantized: the numbers are stored with fewer bits so they fit in less memory with a small loss of quality.
  • Token: a piece of a word. Speed is measured in tokens per second; a person reads about 5 to 8 tokens a second.
  • CPU versus GPU: a GPU does the same maths thousands of times in parallel, so it is many times faster for LLMs. A small model on a 4-vCPU node is slower but works, and costs far less.
  • PersistentVolumeClaim (PVC): a request for a disk that survives pod restarts. We keep the models on it so a restarted pod does not download them again.
  • Requests and limits: what the pod is promised (requests) and the most it may use (limits). Over the memory limit, Kubernetes kills the container: OOMKilled (out of memory).

Before you start

  1. A Google Cloud account (the free trial works) and a project with billing linked.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console).
  3. Set your project and turn on the Kubernetes Engine API. Replace the text in quotes with your own project id:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable container.googleapis.com

Part B: the lab

Step 1: create a small GKE cluster (CPU only) and connect

Why: Ollama runs in a pod. Two e2-standard-4 nodes (4 vCPU, 16 GB memory each), one zone, no GPU. We used this same cluster for four labs in a row, so it was created once:

gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --workload-pool=$PROJECT_ID.svc.id.goog
NAME         LOCATION    STATUS   CURRENT_NODE_COUNT  CURRENT_MASTER_VERSION
lab-cluster  us-east1-b  RUNNING  2                   1.36.4-gke.2377000
NAME          MACHINE_TYPE   DISK_SIZE_GB  INITIAL_NODE_COUNT
default-pool  e2-standard-4  50            2
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b 2>&1 | tail -1
kubectl get nodes
kubeconfig entry generated for lab-cluster.
NAME                                         STATUS   ROLES    AGE   VERSION
gke-lab-cluster-default-pool-27d9254d-1nv1   Ready    <none>   39m   v1.36.4-gke.2377000
gke-lab-cluster-default-pool-27d9254d-rl98   Ready    <none>   39m   v1.36.4-gke.2377000

Step 2: deploy Ollama with a disk for the models

Why: one Deployment with one pod, a 10 GB PVC for the models, and a Service. The pod asks for 3 CPUs and 4 GiB memory and may use up to 6 GiB. strategy: Recreate stops the old pod before starting a new one, because a normal disk can be attached to only one pod at a time. OLLAMA_KEEP_ALIVE=30m keeps a loaded model in memory for 30 minutes so repeat questions are fast. Save this as ollama.yaml:

# Ollama on CPU: one pod, a 10 GB disk for the models, and a service on port 11434
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ollama-models}
spec:
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 10Gi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ollama}
spec:
  replicas: 1
  strategy: {type: Recreate}
  selector: {matchLabels: {app: ollama}}
  template:
    metadata: {labels: {app: ollama}}
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports: [{containerPort: 11434}]
        env: [{name: OLLAMA_KEEP_ALIVE, value: "30m"}]
        resources:
          requests: {cpu: "3", memory: 4Gi}
          limits: {memory: 6Gi}
        readinessProbe: {httpGet: {path: /api/version, port: 11434}, periodSeconds: 5}
        volumeMounts: [{name: models, mountPath: /root/.ollama}]
      volumes:
      - {name: models, persistentVolumeClaim: {claimName: ollama-models}}
---
apiVersion: v1
kind: Service
metadata: {name: ollama}
spec:
  selector: {app: ollama}
  ports: [{port: 11434, targetPort: 11434}]
kubectl create namespace llm
kubectl apply -n llm -f ollama.yaml
kubectl -n llm rollout status deploy/ollama --timeout=300s
kubectl -n llm get pods,pvc
# open the Ollama API on localhost:11434 (keeps running in the background)
nohup kubectl -n llm port-forward svc/ollama 11434:11434 > pf-ollama.log 2>&1 &
sleep 4; curl -s http://localhost:11434/api/version; echo
namespace/llm created
persistentvolumeclaim/ollama-models created
deployment.apps/ollama created
service/ollama created
Waiting for deployment "ollama" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama" successfully rolled out
NAME                          READY   STATUS    RESTARTS   AGE
pod/ollama-85479f4c87-sqfhv   1/1     Running   0          3m2s

NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/ollama-models   Bound    pvc-500eeba1-c6d1-4036-badc-275ceca93a31   10Gi       RWO            standard-rwo   <unset>                 3m2s
{"version":"0.40.16"}
Deploy Ollama
Steps 1 and 2: the pod is Running, the 10 Gi disk is Bound, Ollama 0.40.16 answers

The first start took about 3 minutes: GKE created the disk and the node downloaded the Ollama image (about 1 GB). {"version":"0.40.16"} is Ollama answering on localhost:11434 through kubectl port-forward.

Step 3: download two small open models

Why: ollama pull downloads a model into the pod's disk. We take two: Llama 3.2 1B (Meta) and Qwen 2.5 0.5B (Alibaba), both made to run on small machines. Check each model's licence before you use it in a product.

# download two small open models into the pod's disk (1B and 0.5B parameters)
kubectl -n llm exec deploy/ollama -- ollama pull llama3.2:1b 2>&1 | tail -1
kubectl -n llm exec deploy/ollama -- ollama pull qwen2.5:0.5b 2>&1 | tail -1
kubectl -n llm exec deploy/ollama -- ollama list
success 
success 
NAME            ID              SIZE      MODIFIED               
qwen2.5:0.5b    a8b0c5157701    397 MB    Less than a second ago    
llama3.2:1b     baf6a787fdff    1.3 GB    5 seconds ago

Step 4: ask the model a question

Why: Ollama's /api/generate takes a JSON request: the model name, the prompt, stream: false (one answer at the end instead of word by word), a low temperature for steady answers and num_predict to cap the answer at 120 tokens. The answer JSON also has timing fields, which we turn into tokens per second with jq.

cat > ask.json <<'JSON'
{"model": "llama3.2:1b", "stream": false,
 "prompt": "Explain what a Kubernetes pod is to a beginner in 2 short sentences.",
 "options": {"temperature": 0.2, "num_predict": 120}}
JSON
curl -s http://localhost:11434/api/generate -d @ask.json > answer.json
jq -r '.response' answer.json
jq -r '"tokens out: \(.eval_count), speed: \((.eval_count / (.eval_duration / 1e9)) * 10 | round / 10) tokens per second, total time: \((.total_duration / 1e9) * 10 | round / 10) s (load \((.load_duration / 1e9) * 10 | round / 10) s)"' answer.json
A Kubernetes pod is a basic execution unit in Kubernetes that represents a single instance of a container, consisting of one or more containers that run together. Pods are the building blocks of Kubernetes deployments and can be scaled up or down based on demand, allowing for efficient resource utilization and high availability.
tokens out: 58, speed: 10.2 tokens per second, total time: 9.7 s (load 2.9 s)
First answer
Steps 3 and 4: two models downloaded; Llama 3.2 1B answers in 9.7 s at 10.2 tokens per second

A correct, readable answer from a model running on your own CPU node. 10.2 tokens a second is about reading speed. load 2.9 s is the time to load the model from disk into memory the first time.

Step 5: measure the speed of both models

Why: model size is the main speed lever on CPU. We ask both models the same question twice.

# same question to both models, 2 rounds each: compare speed on CPU
for m in llama3.2:1b qwen2.5:0.5b; do
  for r in 1 2; do
    jq -n --arg m "$m" '{model: $m, stream: false, prompt: "List 3 benefits of containers in one line each.", options: {temperature: 0.2, num_predict: 80}}' > q.json
    curl -s http://localhost:11434/api/generate -d @q.json | jq -r --arg m "$m" --arg r "$r" '"\($m) round \($r): \(.eval_count) tokens, \((.eval_count / (.eval_duration / 1e9)) * 10 | round / 10) tokens/s, \((.total_duration / 1e9) * 10 | round / 10) s total"'
  done
done
# how much CPU and memory the pod used (metrics from the last minute)
kubectl -n llm top pod
llama3.2:1b round 1: 55 tokens, 10.3 tokens/s, 5.8 s total
llama3.2:1b round 2: 61 tokens, 10.3 tokens/s, 6.1 s total
qwen2.5:0.5b round 1: 80 tokens, 23.7 tokens/s, 5.5 s total
qwen2.5:0.5b round 2: 80 tokens, 23.7 tokens/s, 3.4 s total
NAME                      CPU(cores)   MEMORY(bytes)   
ollama-85479f4c87-sqfhv   368m         277Mi
Speed test
Llama 3.2 1B: about 10 tokens per second. Qwen 2.5 0.5B: about 24 tokens per second
ModelSize on diskSpeed on 3 vCPU (our test)Good for
llama3.2:1b1.3 GBabout 10 tokens per secondshort answers, summaries, classification
qwen2.5:0.5b397 MBabout 24 tokens per secondvery simple tasks, fast tests

The second Qwen round was faster in total (3.4 s against 5.5 s) because the model was already in memory. kubectl top showed the pod using little CPU at that moment, because metrics are an average over the last minute and the pod is idle between questions. Watch the cost: the node costs the same whether the model is busy or idle, so a self-hosted model is cheap only when it is used a lot; for a few questions a day, a pay-per-token API is cheaper. To serve more users, add replicas, which each need their own CPU and memory (and their own copy of the models, because a normal disk attaches to one pod).

Break it and fix it: too little memory

We set the memory limit to 512 MiB, far less than the 1B model needs, and ask the same question again.

# Break it: squeeze the memory limit to 512 MiB (the 1B model needs more than that)
kubectl -n llm set resources deploy/ollama --limits=memory=512Mi --requests=memory=256Mi
kubectl -n llm rollout status deploy/ollama --timeout=180s
pkill -f "[p]ort-forward svc/ollama"; nohup kubectl -n llm port-forward svc/ollama 11434:11434 > pf-ollama.log 2>&1 &
sleep 4
curl -s -m 120 http://localhost:11434/api/generate -d @ask.json | head -c 300; echo
sleep 5
kubectl -n llm get pods
kubectl -n llm get pod -l app=ollama -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.reason}'; echo
🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโ‚น10,620See the training →
deployment.apps/ollama resource requirements updated
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama" successfully rolled out

NAME                      READY   STATUS    RESTARTS     AGE
ollama-7bbdc9cd9f-plqx9   1/1     Running   1 (7s ago)   19s
OOMKilled
OOMKilled
The request returns nothing, the pod restarts and the reason is OOMKilled

The answer is empty (the connection dropped), the pod shows RESTARTS 1 and the last termination reason is OOMKilled: Ollama tried to load the model into memory, went over 512 MiB, and the kernel killed it. Fix: give it enough memory again.

# Fix it: give the pod enough memory again
kubectl -n llm set resources deploy/ollama --limits=memory=6Gi --requests=memory=4Gi
kubectl -n llm rollout status deploy/ollama --timeout=180s
pkill -f "[p]ort-forward svc/ollama"; nohup kubectl -n llm port-forward svc/ollama 11434:11434 > pf-ollama.log 2>&1 &
sleep 4
curl -s http://localhost:11434/api/generate -d @ask.json | jq -r '.response'
kubectl -n llm get pods
# the models were still on the disk (PersistentVolumeClaim), so nothing had to be downloaded again
kubectl -n llm exec deploy/ollama -- ollama list
deployment.apps/ollama resource requirements updated
Waiting for deployment spec update to be observed...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "ollama" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama" successfully rolled out
A Kubernetes pod is a basic execution unit in a Kubernetes cluster, consisting of one or more containers that run together on a single host, providing a single IP address and network identity. Pods are the building blocks of Kubernetes applications, and they can be scaled up or down based on demand, allowing for efficient resource utilization and high availability.
NAME                      READY   STATUS    RESTARTS   AGE
ollama-85479f4c87-22l98   1/1     Running   0          2m29s
NAME            ID              SIZE      MODIFIED      
qwen2.5:0.5b    a8b0c5157701    397 MB    4 minutes ago    
llama3.2:1b     baf6a787fdff    1.3 GB    4 minutes ago
Fixed
With 4 Gi requested and 6 Gi allowed, the answer comes back; the models were still on the disk

The models were still on the PVC, so nothing was downloaded again. Rule of thumb for CPU serving: memory limit of at least 2 to 3 times the model file size.

POC: prove it works

  • Ollama runs as a normal Kubernetes Deployment, with its models on a persistent disk.
  • Llama 3.2 1B answered a real question at about 10 tokens per second on CPU; Qwen 2.5 0.5B at about 24.
  • Too little memory gives OOMKilled; the right limits fix it, with no new download.
  • No GPU, no API key, no data leaves your cluster.

Clean up (delete everything)

Why: the PVC is a real Compute Engine disk that adds to your bill even when no pod uses it. Deleting the namespace deletes the PVC, and GKE then deletes the disk.

pkill -f "[p]ort-forward svc/ollama"
kubectl delete namespace llm --wait=true
# the model disk was a Compute Engine persistent disk; deleting the claim deletes it
sleep 20
echo "namespaces left from this lab: $(kubectl get ns llm --no-headers 2>/dev/null | wc -l)"
echo "model disks left: $(gcloud compute disks list --filter="name~^pvc-" --format="value(name)" | wc -l)"
namespace "llm" deleted
namespaces left from this lab: 0
model disks left: 0

If you are not doing another Kubernetes lab now, delete the cluster (we did this right after this lab, our fourth on it):

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet 2>&1 | tail -1
echo "clusters left: $(gcloud container clusters list --format="value(name)" | wc -l)"
echo "cluster disks left: $(gcloud compute disks list --filter="name~^gke-lab-cluster OR name~^pvc-" --format="value(name)" | wc -l)"
echo "load balancers left: $(gcloud compute forwarding-rules list --format="value(name)" | wc -l)"
Deleted [https://container.googleapis.com/v1/projects/your-project-id/zones/us-east1-b/clusters/lab-cluster].
clusters left: 0
cluster disks left: 0
load balancers left: 0
Clean up
0 lab namespaces, 0 model disks, 0 clusters, 0 disks, 0 load balancers

FAQ

Can I run bigger models like 7B or 8B on CPU? They run, but slowly (often 2 to 5 tokens per second on 4 vCPUs) and need 8 GB or more memory. For those, use a GPU node pool. GPU labs are a separate post.

Is a self-hosted model private? The prompts and answers stay in your cluster. That is a common reason companies self-host. You still own security: keep the service internal, as here (ClusterIP and port-forward only).

Ollama or vLLM? Ollama is the easiest way to start and runs well on CPU. vLLM is built for high-throughput GPU serving in production.

How do I call it from another app in the cluster? Use the service address http://ollama.llm:11434/api/generate (service ollama, namespace llm).

Does it have an OpenAI-style API? Yes, Ollama also answers on /v1/chat/completions. We tested only /api/generate in this lab.

Why is the first answer slower? The model is loaded from disk into memory first (load_duration). With OLLAMA_KEEP_ALIVE it stays loaded.

Troubleshooting

You seeWhyFix
Pod Pending with "Insufficient cpu" or "memory"The node is too small for the requestsUse e2-standard-4 nodes or lower the requests
Pod restarts, reason OOMKilledMemory limit below what the model needsRaise the limit (2 to 3 times the model size)
Multi-Attach error for the volumeTwo pods want the same ReadWriteOnce diskUse strategy: Recreate and one replica per disk
connection refused on localhost:11434The port-forward stopped (for example after a pod restart)Start kubectl -n llm port-forward svc/ollama 11434:11434 again
model "x" not foundThe model was not pulledkubectl -n llm exec deploy/ollama -- ollama pull x
Very slow answersBig model or few CPUsUse a smaller model or give the pod more CPUs

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Ray on Kubernetes with KubeRay: Distributed Python for AI⏱ 10 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →