CareerByteCode
ByteBlogby CareerByteCode

Ray on Kubernetes with KubeRay: Distributed Python for AI

What you build: a Ray cluster on GKE with KubeRay, parallel Python tasks, a stuck oversized task and a self-cleaning RayJob. Every step tested.

Ray on Kubernetes with KubeRay: Distributed Python for AI

✍️ Translate this blog into Hindi, Tamil or Telugu

Ray lets you take a normal Python function and run many copies of it across many machines with two small changes. KubeRay runs Ray on Kubernetes, so the same cluster that runs your apps can also run your data and AI jobs. In this lab you install KubeRay on GKE, start a small Ray cluster, run parallel Python on it, see what happens when a task asks for more than the cluster has, and run a self-cleaning RayJob.

Who it is for: Python developers and platform engineers who are curious how AI teams run distributed work on Kubernetes. No GPU is needed: everything here runs on CPUs. Time: about 1 hour. Cost: a small GKE cluster for about 15 minutes, covered by the free trial credit. Delete it at the end.

Every command below was run for real by the CareerByteCode Team, and every output is the real result we got, including one result that surprised us. We typed the commands in a Cloud Shell-like Linux VM (Debian with gcloud, kubectl and helm); Cloud Shell in your browser has the same tools.

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →

Part A: Ray and KubeRay in plain English

Python normally runs on one CPU at a time. Ray turns a function into a task that Ray can send to any CPU in a Ray cluster, and gives you the result back later.

How KubeRay runs Ray on Kubernetes
The KubeRay operator turns a RayCluster object into a head pod and worker pods; your Python sends tasks to them

Words you will use:

  • Ray cluster: one head (it keeps track of everything and schedules work) and some workers (they run tasks). Here: 1 head and 2 workers, 1 CPU each.
  • Task: a function marked with @ray.remote (or ray.remote(f)). Calling f.remote(x) starts it somewhere in the cluster and returns a ticket; ray.get(ticket) waits for the answer.
  • Resources: each task says what it needs, for example num_cpus=1. Ray only starts a task where that much is free.
  • KubeRay operator: a Kubernetes controller that understands three new object types: RayCluster (a long-running cluster), RayJob (a cluster made for one job, then removed) and RayService (serving models).
  • Real-life example: Ray is a restaurant kitchen. The head chef (head) takes orders and gives them to cooks (workers). An order that needs 16 cooks waits forever in a kitchen with 3.

Before you start

  1. A Google Cloud project with billing linked (the free trial works) and Cloud Shell open.
  2. One small GKE cluster. We created it once for this batch of Kubernetes labs:
gcloud container clusters create lab-cluster \
  --zone=us-east1-b --num-nodes=2 --machine-type=e2-standard-4 \
  --disk-type=pd-standard --disk-size=50 --release-channel=regular \
  --no-enable-autoupgrade --enable-dataplane-v2

Part B: the lab

Step 1: install the KubeRay operator

Why: the operator watches for Ray objects and creates the pods for them. We pin chart version 1.2.2.

# Install the KubeRay operator with Helm (it adds the RayCluster, RayJob and RayService resource types)
gcloud container clusters get-credentials lab-cluster --zone=us-east1-b > /dev/null 2>&1
helm repo add kuberay https://ray-project.github.io/kuberay-helm/ > /dev/null 2>&1 && helm repo update kuberay > /dev/null 2>&1
helm install kuberay-operator kuberay/kuberay-operator --version 1.2.2 -n ray-system --create-namespace --wait 2>&1 | grep -E "STATUS|REVISION"
kubectl get crd -o name | grep ray.io
STATUS: deployed
REVISION: 1
customresourcedefinition.apiextensions.k8s.io/rayclusters.ray.io
customresourcedefinition.apiextensions.k8s.io/rayjobs.ray.io
customresourcedefinition.apiextensions.k8s.io/rayservices.ray.io

The three lines show the new object types the operator added: RayCluster, RayJob and RayService.

Step 2: start a small Ray cluster

Why: one RayCluster object describes the whole cluster. rayStartParams: {num-cpus: "1"} tells Ray how many CPUs each pod offers, and the resources block tells Kubernetes the same thing. Keep the two equal. Save this as raycluster.yaml:

# A small Ray cluster on Kubernetes: 1 head and 2 workers, CPU only (1 CPU each), managed by the KubeRay operator
apiVersion: ray.io/v1
kind: RayCluster
metadata: {name: demo, namespace: ray-demo}
spec:
  rayVersion: "2.40.0"
  headGroupSpec:
    rayStartParams: {num-cpus: "1", dashboard-host: "0.0.0.0"}
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:2.40.0-py311
          resources:
            requests: {cpu: "1", memory: "2Gi"}
            limits: {cpu: "1", memory: "2Gi"}
  workerGroupSpecs:
  - groupName: cpu
    replicas: 2
    minReplicas: 2
    maxReplicas: 2
    rayStartParams: {num-cpus: "1"}
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray:2.40.0-py311
          resources:
            requests: {cpu: "1", memory: "2Gi"}
            limits: {cpu: "1", memory: "2Gi"}
# A Ray cluster: 1 head and 2 workers, 1 CPU each (the image is large, so the first start takes a few minutes)
kubectl create namespace ray-demo > /dev/null && kubectl apply -f raycluster.yaml
for i in $(seq 1 60); do S=$(kubectl -n ray-demo get raycluster demo -o jsonpath='{.status.state}'); [ "$S" = ready ] && break; sleep 10; done; echo "raycluster state: $S"
kubectl -n ray-demo get pods -o custom-columns=POD:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName --no-headers | sed -E 's/gke-lab-cluster-default-pool-//'
raycluster.ray.io/demo created
raycluster state: ready
demo-cpu-worker-5n9cn   Running   dae4ce23-mr48
demo-cpu-worker-tg8gz   Running   dae4ce23-cp97
demo-head-shgz5         Running   dae4ce23-cp97
The Ray cluster
Steps 1 and 2: the operator, then a head and two workers spread over the two nodes

Step 3: run parallel Python on it

Why: this is what Ray is for. pi.py estimates pi by throwing 6 million random points at a square and counting how many land inside a circle. It does the job once on one CPU, then as 6 Ray tasks of 1 million points each. Save it as pi.py:

# pi.py: estimate pi with random points. First on one CPU, then split into 6 Ray tasks across the cluster's 3 CPUs.
import random, time
import ray

N = 6_000_000


def count_inside(n, seed):
    rnd = random.Random(seed); inside = 0
    for _ in range(n):
        x, y = rnd.random(), rnd.random()
        inside += x * x + y * y <= 1.0
    return inside


ray.init(address="auto")
print("cluster CPUs:", int(ray.cluster_resources().get("CPU", 0)), "| nodes:", len([n for n in ray.nodes() if n["Alive"]]))

t = time.time(); serial = count_inside(N, 1); ts = time.time() - t
print(f"serial:   pi ~ {4 * serial / N:.5f} in {ts:.1f} s")

remote_count = ray.remote(num_cpus=1)(count_inside)
t = time.time(); parts = ray.get([remote_count.remote(N // 6, s) for s in range(6)]); tp = time.time() - t
print(f"parallel: pi ~ {4 * sum(parts) / N:.5f} in {tp:.1f} s with 6 tasks ({ts / tp:.1f}x faster)")

We copy the script into the head pod and run it there (ray.init(address="auto") connects to the cluster it runs in):

# Run Python on the cluster: estimate pi on one CPU, then as 6 Ray tasks spread over 3 CPUs
HEAD=$(kubectl -n ray-demo get pod -l ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')
kubectl -n ray-demo cp pi.py $HEAD:/tmp/pi.py && kubectl -n ray-demo exec $HEAD -- python /tmp/pi.py 2>&1 | grep -vE "^20[0-9]{2}-|INFO|connecting to existing"
cluster CPUs: 3 | nodes: 3
serial:   pi ~ 3.14206 in 1.7 s
parallel: pi ~ 3.14136 in 2.3 s with 6 tasks (0.7x faster)

Both answers are close to 3.14159, so the parallel version works. But look at the time: the parallel run was slower (0.7x). That is a real result, and a useful lesson. Each Ray task has a fixed cost: Ray starts a worker process, ships the function and its arguments, and collects the result. A job that takes 1.7 seconds on one CPU is too small to win back that cost on a fresh 3-CPU cluster. Ray pays off when each task does seconds or minutes of work (training a model per data slice, processing a file per task), not milliseconds.

Break it: a task the cluster can never run

We ask Ray for a task that needs 16 CPUs on our 3-CPU cluster and wait 20 seconds. Save this as too_big.py:

# too_big.py: ask Ray for a task that needs 16 CPUs on a 3-CPU cluster, and wait 20 seconds for it
import ray
ray.init(address="auto")


@ray.remote(num_cpus=16)
def huge():
    return "done"


ready, not_ready = ray.wait([huge.remote()], timeout=20)
print("finished:", len(ready), "| still waiting:", len(not_ready))
print("cluster CPUs:", int(ray.cluster_resources().get("CPU", 0)), "| a task asking for 16 CPUs can never start here")
# Break it: a task that asks for 16 CPUs on a 3-CPU cluster
HEAD=$(kubectl -n ray-demo get pod -l ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')
kubectl -n ray-demo cp too_big.py $HEAD:/tmp/too_big.py && kubectl -n ray-demo exec $HEAD -- python /tmp/too_big.py 2>&1 | grep -vE "^20[0-9]{2}-|INFO|connecting to existing" | grep -E "finished|cluster CPUs|cannot be scheduled|infeasible|Warning" | cut -c1-220 | head -4
finished: 0 | still waiting: 1
cluster CPUs: 3 | a task asking for 16 CPUs can never start here

There is no error and no crash: after 20 seconds the task has still not started, and it never will, because no node has 16 CPUs. In a real team this looks like "my job is stuck". The fix is to size the request to the cluster (or let the cluster grow with Ray's autoscaler, which we did not set up here).

Break it
Step 3 and Break it: pi in parallel, then a 16-CPU task that never starts

Fix it: a RayJob that brings its own cluster

🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโ‚น10,620See the training →

Why: a long-running RayCluster keeps costing while nobody uses it. A RayJob creates a Ray cluster sized for one job, runs the entrypoint, and with shutdownAfterJobFinishes: true deletes the cluster 30 seconds after the job ends. Save this as rayjob.yaml (the >- keeps the long command on one line inside YAML):

# A RayJob: KubeRay creates a temporary Ray cluster, runs the entrypoint, and deletes the cluster when it finishes
apiVersion: ray.io/v1
kind: RayJob
metadata: {name: squares, namespace: ray-demo}
spec:
  entrypoint: >-
    python -c "import ray; ray.init(); f = ray.remote(lambda x: x * x); print('sum of squares 0..99 =', sum(ray.get([f.remote(i) for i in range(100)])))"
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 30
  rayClusterSpec:
    rayVersion: "2.40.0"
    headGroupSpec:
      rayStartParams: {num-cpus: "1"}
      template:
        spec:
          containers:
          - name: ray-head
            image: rayproject/ray:2.40.0-py311
            resources:
              requests: {cpu: "1", memory: "2Gi"}
              limits: {cpu: "1", memory: "2Gi"}
# Fix the habit: size the job to the cluster. A RayJob creates its own Ray cluster, runs, and cleans up
kubectl apply -f rayjob.yaml
for i in $(seq 1 60); do S=$(kubectl -n ray-demo get rayjob squares -o jsonpath='{.status.jobStatus}'); [ "$S" = SUCCEEDED ] || [ "$S" = FAILED ] && break; sleep 10; done; echo "rayjob status: $S"
kubectl -n ray-demo logs job/squares 2>/dev/null | grep "sum of squares" | head -1
rayjob.ray.io/squares created
rayjob status: SUCCEEDED
sum of squares 0..99 = 328350

The sum of the squares of 0 to 99 is 328,350, so the 100 tasks ran and the answer came back. KubeRay then removed the job's cluster by itself.

POC: prove it works

  • KubeRay created a head and two workers from one RayCluster object (Step 2).
  • Ray ran 6 tasks across the cluster and got the right answer for pi (Step 3).
  • An oversized task waited instead of failing (Break it), and a RayJob ran 100 tasks and cleaned up after itself (Fix it).

Clean up (delete everything)

Why: Ray pods hold CPUs and memory while they exist. We delete the lab namespace, the operator and the Ray object types.

# Clean up: the Ray clusters, the jobs and the operator
kubectl delete namespace ray-demo --wait=true 2>&1 | tail -1
helm uninstall kuberay-operator -n ray-system 2>&1 | tail -1; kubectl delete namespace ray-system --wait=true 2>&1 | tail -1
kubectl get crd -o name | grep ray.io | xargs -r kubectl delete > /dev/null 2>&1; echo "ray CRDs left: $(kubectl get crd -o name | grep -c ray.io)"
namespace "ray-demo" deleted
release "kuberay-operator" uninstalled
namespace "ray-system" deleted
ray CRDs left: 0
Clean up
The RayJob result, then 0 Ray object types left

When you have finished all your Kubernetes labs, delete the cluster:

gcloud container clusters delete lab-cluster --zone=us-east1-b --quiet

FAQ

Do I need GPUs for Ray? No. Ray schedules CPUs, GPUs and custom resources the same way. This lab is CPU only; with GPU nodes you would ask for num_gpus=1 in a task.

Why was the parallel version slower? Fixed costs per task (worker start, sending the function, collecting the result) were bigger than the work itself. Run it a second time on a warm cluster, give each task more work, or use fewer, larger tasks.

RayCluster or RayJob? RayCluster for interactive work and notebooks; RayJob for batch jobs in pipelines, because it cleans up after itself.

What is a RayService? It runs Ray Serve, Ray's model-serving library, and can upgrade a serving cluster without downtime. We did not test it here.

Can Ray add workers by itself? Yes, with enableInTreeAutoscaling: true in the RayCluster and min and max replicas per worker group. We did not test autoscaling in this lab.

Troubleshooting

You seeWhyFix
The RayCluster stays not ready for minutesThe Ray image is large and is downloadingWait; the second start on the same node is fast
Pods stay PendingThe nodes do not have the CPU or memory the pods requestLower requests or add nodes
A task never starts, no errorIt asks for more CPU, GPU or memory than any node offersCheck ray.cluster_resources() and size the task to fit
error converting YAML to JSON on a RayJobA long entrypoint with : inside breaks YAMLUse entrypoint: >- and put the command on the next line, as above
Parallel code is slowerTasks are too smallGive each task more work, use fewer tasks

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →KServe Hands-On: Serverless Model Serving on Kubernetes⏱ 11 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →