Kubernetes for AI Workloads: GPU Infrastructure Without the GPU

Kubernetes for AI workloads looks a lot like the Kubernetes you already run. You still describe what you want in manifests, the scheduler still finds a place for it, and something still has to notice when it breaks. The differences sit in a few specific places.

Those places are scheduling, serving and cost. There's a CPU-only exercise at the end that you can run on a laptop. Documentation details are current as of October 2026.

Why Kubernetes keeps showing up

Kubernetes works from declared desired state. You say how many copies of a service should run and what each one needs, and the control plane keeps reality close to that. AI workloads need the same things: model servers that stay up, batch jobs that finish, and capacity that follows demand.

Much of your DevOps toolkit carries over.

Two things are new. The hardware is scarce and expensive, and what you deploy behaves in ways you can't fully predict. The next sections show where that matters.

Scheduling: where GPUs change the rules

Kubernetes exposes a GPU as a countable resource once the vendor's driver and device plugin are installed on the node. A container asks for it like CPU or memory, with one catch. GPUs go in limits only. You can add a request too, but it has to equal the limit, and a request without a limit isn't allowed.

resources:
  limits:
    nvidia.com/gpu: 1

Clusters rarely have one kind of GPU. The usual setup labels nodes by accelerator type, matches workloads to them with node selectors or affinity, and taints GPU nodes so ordinary pods don't take them by accident. Node Feature Discovery can apply the labels for you.

Dynamic Resource Allocation, now stable, changes the request itself. Instead of asking for one GPU, a workload creates a claim that selects a device by its attributes from a device class, and the scheduler finds a node that fits. A plain count can't express that.

Groups are the next problem. A distributed training job needs all its workers at once, but the default scheduler places pods one at a time. A job can grab some of the GPUs, then sit on them while it waits for the rest, and other jobs starve.

Say you book a table for ten and the restaurant seats you five at a time. Half your group is at the bar while the other half eats. Gang scheduling is the rule that all ten get a table or nobody sits. Kubernetes now has it built in through its workload and pod group APIs. It's beta and off by default, so you have to switch it on. Separate batch schedulers have filled the gap for years, and many clusters still use them.

Sharing capacity between teams is the third problem, and Kueue handles it on top of the scheduler. You describe the hardware as resource flavours, give each team a local queue, and a cluster queue enforces the quotas. Teams in the same cohort can borrow each other's unused quota, and the owner can take it back through preemption when needed.

Serving: a familiar service with an unusual start

Serving a trained model looks like running any long-lived service: a Kubernetes Service, requests in, responses out. Start-up and routing are where it differs.

A model server can need minutes to load its weights before it answers anything. A liveness probe that fires too early kills the container before it's ready, and the restarts never stop. A startup probe fixes that by holding the other probes off until the container has started. Section 7 uses one.

Routing is the other difference. Ordinary load balancing treats replicas as equal, but model replicas vary in how busy they are and what they've cached. The Gateway API Inference Extension is an open-source answer. It groups model-server pods into an inference pool and picks an endpoint using signals like queue depth and cache usage. It can also route by the model name in the request and drop low-priority traffic first under pressure. Check its maturity and your platform's support before you depend on it.

Scaling and cost: the GPU is the bill

An idle GPU is expensive, so cost control belongs in the design from the start.

Quotas come first. A ResourceQuota can cap the GPUs a namespace may request, using the requests. prefix on the resource name. A PriorityClass lets production inference preempt lower-priority training pods, so a customer-facing service doesn't wait behind a long experiment. Vendor features can let several workloads share one physical GPU, and spot capacity suits jobs that checkpoint often enough to survive interruption.

Tune quotas by watching utilisation. Too tight and hardware sits idle. Too loose and borrowed capacity gets reclaimed in the middle of a run. There's no formula.

Production realities

The dull problems cause the incidents. One practitioner write-up reports that a failing GPU can leave a container looking healthy to a liveness probe, because the process is alive while the device underneath is not. Its suggested fix is to export accelerator metrics and use node problem detection to taint bad nodes. I'd test that on your own platform before trusting it, but it makes sense, since ordinary health checks were designed for CPU workloads.

Storage is the other recurring bottleneck. A slow filesystem leaves an expensive GPU waiting for data. Pod disruption budgets and termination grace periods also interact with long jobs in ways that surprise people.

Your DevOps habits count for a lot here. Dashboards, alerts and careful rollouts are the daily work.

A reference pattern

This is a general pattern and doesn't describe any specific system.

Traffic enters through a gateway and reaches an API service, which calls the model servers. Batch work comes in through a queue. Both run on GPU nodes under shared quotas and priorities, and monitoring watches all of it.

How to practise without a GPU cluster

You don't need a GPU to learn most of the Kubernetes side. This reference exercise moves a small two-service setup from Docker Compose to a local cluster, using a tiny model that runs on CPU. Ignore the quality of its answers and watch how Kubernetes behaves around the model.

You'll need Docker, kubectl and minikube.

Note: Pin image tags in real use. The samples below leave tags off to stay readable and to avoid going stale.

The steps:

1. Step 1: Run the setup in Docker Compose.
2. Step 2: Translate each service into a Deployment and a Service.
3. Step 3: Add resource requests and limits.
4. Step 4: Add health probes, including a startup probe.
5. Step 5: Move configuration and secrets out of the image.
6. Step 6: Add autoscaling and watch it respond.
7. Step 7: Add monitoring.

Step 1: Run it in Docker Compose

Create a small API that forwards a prompt to a local model server. It uses only the Python standard library, and the /work endpoint does a little arithmetic so there's something to burn CPU when you reach autoscaling.

import json
import os
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

MODEL_URL = os.environ.get("MODEL_URL", "http://model-server:11434")
MODEL_NAME = os.environ.get("MODEL_NAME", "tinyllama")
API_TOKEN = os.environ.get("API_TOKEN", "")


class Handler(BaseHTTPRequestHandler):
    def _send(self, code, body):
        data = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        if self.path == "/health":
            self._send(200, {"status": "ok"})
        elif self.path == "/work":
            total = sum(i * i for i in range(200_000))
            self._send(200, {"result": total})
        else:
            self._send(404, {"error": "not found"})

    def do_POST(self):
        if self.path != "/ask":
            return self._send(404, {"error": "not found"})
        if API_TOKEN and self.headers.get("X-Token") != API_TOKEN:
            return self._send(401, {"error": "unauthorized"})
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length) or b"{}")
        request = urllib.request.Request(
            f"{MODEL_URL}/api/generate",
            data=json.dumps(
                {"model": MODEL_NAME, "prompt": body.get("prompt", ""), "stream": False}
            ).encode(),
            headers={"Content-Type": "application/json"},
        )
        with urllib.request.urlopen(request, timeout=120) as response:
            answer = json.loads(response.read())
        self._send(200, {"answer": answer.get("response", "")})


ThreadingHTTPServer(("0.0.0.0", 8000), Handler).serve_forever()

Save it as api/app.py with a minimal Dockerfile.

FROM python:3-slim
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]

Then describe both services in compose.yaml.

services:
  api:
    build: ./api
    ports:
      - "8000:8000"
    environment:
      MODEL_URL: http://model-server:11434
      MODEL_NAME: tinyllama
      API_TOKEN: ${API_TOKEN}
    depends_on:
      - model-server
  model-server:
    image: ollama/ollama
    volumes:
      - models:/root/.ollama
volumes:
  models:

Bring it up and pull the small model once.

export API_TOKEN=change-me
docker compose up -d
docker compose exec model-server ollama pull tinyllama

This is your starting point. The API reaches the model server by service name, and the model files live on a volume so they survive restarts.

Step 2: Translate each service into a Deployment and a Service

Start the cluster and build the API image into it.

minikube start --cpus=4 --memory=6g
minikube image build -t ai-api:dev ./api

Each Compose service becomes a Deployment, which keeps the right number of copies running, plus a Service, which gives it a stable name. The model files move to a PersistentVolumeClaim.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-data
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: model-server
  template:
    metadata:
      labels:
        app: model-server
    spec:
      containers:
        - name: ollama
          image: ollama/ollama
          ports:
            - containerPort: 11434
          volumeMounts:
            - name: model-data
              mountPath: /root/.ollama
      volumes:
        - name: model-data
          persistentVolumeClaim:
            claimName: model-data
---
apiVersion: v1
kind: Service
metadata:
  name: model-server
spec:
  selector:
    app: model-server
  ports:
    - port: 11434
      targetPort: 11434
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ai-api
  template:
    metadata:
      labels:
        app: ai-api
    spec:
      containers:
        - name: api
          image: ai-api:dev
          ports:
            - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: ai-api
spec:
  selector:
    app: ai-api
  ports:
    - port: 80
      targetPort: 8000

The model server uses the Recreate strategy because a ReadWriteOnce volume is generally attached to one node at a time. A rolling update would start the new pod while the old one still holds the volume, and the two would compete for it.

Apply the file, then pull the model once the pod is running.

kubectl apply -f k8s/all.yaml
kubectl exec deploy/model-server -- ollama pull tinyllama

Step 3: Add resource requests and limits

Without requests the scheduler has nothing to reason about. It's the CPU and memory version of the GPU limit from earlier. Add these under each container.

# model-server container
resources:
  requests:
    cpu: "1"
    memory: 2Gi
  limits:
    memory: 4Gi

# api container
resources:
  requests:
    cpu: 100m
    memory: 64Mi
  limits:
    memory: 128Mi

The API's CPU request matters later, because autoscaling measures usage as a percentage of it.

Step 4: Add health probes, including a startup probe

A model server can take a while to become ready. A startup probe gives it time before the other probes begin, and if it never succeeds the container is restarted. Add this to the model-server container.

startupProbe:
  httpGet:
    path: /
    port: 11434
  periodSeconds: 10
  failureThreshold: 30
readinessProbe:
  httpGet:
    path: /
    port: 11434
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /
    port: 11434
  periodSeconds: 20

That allows up to five minutes. Add a simpler pair for the API, pointing at /health on port 8000. Then remove the startup probe and watch a slow start turn into a restart loop.

Step 5: Move configuration and secrets out of the image

The Compose file kept the model address and token in environment variables. Kubernetes splits them: ordinary settings go in a ConfigMap and sensitive values in a Secret.

kubectl create secret generic api-secrets --from-literal=API_TOKEN=change-me

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
data:
  MODEL_URL: http://model-server:11434
  MODEL_NAME: tinyllama

Reference both from the API container.

envFrom:
  - configMapRef:
      name: api-config
env:
  - name: API_TOKEN
    valueFrom:
      secretKeyRef:
        name: api-secrets
        key: API_TOKEN

Now the same image can move between environments with different settings, and the token never lands in a Git repository or a container layer.

Step 6: Add autoscaling and watch it respond

Horizontal autoscaling needs the metrics server, which minikube can enable.

minikube addons enable metrics-server
kubectl autoscale deployment ai-api --cpu=50% --min=1 --max=5

Now generate load against /work and watch the replica count.

kubectl run load-generator --rm -i --tty --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://ai-api/work; done"
kubectl get hpa ai-api --watch

Scaling the API is the easy half. Scaling a model server on CPU use would load a full copy of the model into every new replica, so real model servers usually scale on signals like request queue depth, and on GPUs they're limited by how many devices you have. This step shows the mechanism and where it stops.

Test the whole chain from your machine once everything is up.

kubectl port-forward svc/ai-api 8080:80 &
curl -s localhost:8080/ask -H "X-Token: change-me" -d '{"prompt":"Say hello in five words."}'

Start-Process kubectl -ArgumentList "port-forward svc/ai-api 8080:80"
Invoke-RestMethod -Uri http://localhost:8080/ask -Method Post -Headers @{"X-Token"="change-me"} -Body '{"prompt":"Say hello in five words."}'

Step 7: Add monitoring

Start with what the cluster already knows.

kubectl top pods
kubectl describe pod -l app=model-server

Then decide what you'd alert on in production. For this setup that means request latency, restarts, memory against its limit, and time spent not ready. Write those down. On GPU hardware the list gains accelerator utilisation and health, and the habit stays the same.

Use this list to check your progress.

• Compose setup answers a prompt locally
• Both services run as Deployments with Services
• Resource requests and limits are set
• The startup probe lets a slow model load without restarts
• Configuration lives in a ConfigMap and the token in a Secret
• The autoscaler adds replicas under load and removes them afterwards
• You can name the four signals you'd alert on

Where to go next

Once this feels comfortable, the GPU material above stops being abstract. You'll recognise the scheduler, the quotas and the probes. What changes is the hardware underneath and the price of mistakes.

If you try it and get stuck, or want to compare notes on moving a proof of concept toward production, message me on LinkedIn and tell me where you got to. I'm happy to look at your manifests with you.

If you want a structured way to learn Kubernetes itself, CareerByteCode also has a Kubernetes Bootcamp.

Checked against the Kubernetes documentation in October 2026. The workload and pod group APIs that provide gang scheduling were beta and disabled by default at that time, and they require a feature gate. Check the current documentation for your cluster's release.