AI Infrastructure Engineer Interview Questions with Labs
What you build: 5 AI infrastructure interview answers proved on a CPU VM: model memory, batching, quantization, CPU throttling and model storage speed. Every step tested.

✍️ Translate this blog into Hindi, Tamil or Telugu
AI infrastructure interviews ask engineering questions, not model trivia: how much memory does a model need, why do inference servers batch, what does quantization buy, why is my container slow, where should model files live? You can answer them with numbers you measured yourself, on a CPU, in minutes. In this post, every answer is proved with a small, real experiment on a Google Cloud VM (2 vCPUs, 8 GB memory, no GPU).
Who it is for: engineers moving into AI platform, MLOps or ML infrastructure roles. Time: about 45 minutes. Cost: one small VM and a temporary bucket for about 10 minutes, covered by the free trial credit. No GPU or TPU is used.
Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. The VM was created with gcloud from a Cloud Shell-like Linux machine; the experiments ran on the VM over SSH.
The lab behind the answers
All experiments are in one small Python file that uses only numpy (a fast library for arrays and matrix maths). A neural network layer is, at its core, a matrix multiply, so numpy is enough to show how the hardware behaves. Save it as infra.py:
# infra.py <question>: small, real measurements behind common AI infrastructure interview answers (CPU only, numpy).
import sys, time
import numpy as np
q = sys.argv[1]
if q == "memory":
# weights only: parameters x bytes per parameter; plus the KV cache a chat model keeps per token
for name, params in (("7B model", 7e9), ("13B model", 13e9), ("70B model", 70e9)):
print(f"{name:<10} fp32 {params * 4 / 1e9:6.0f} GB | fp16/bf16 {params * 2 / 1e9:5.0f} GB | int8 {params / 1e9:5.0f} GB | int4 {params / 2 / 1e9:5.1f} GB")
layers, kv_heads, head_dim, tokens, users = 32, 8, 128, 8192, 16 # a typical 7-8B model with grouped-query attention
kv = 2 * layers * kv_heads * head_dim * 2 * tokens * users # keys + values, fp16
print(f"KV cache for {users} users x {tokens} tokens: {kv / 1e9:.1f} GB more (this is why long context needs big GPUs)")
elif q == "batching":
# one "layer" of a model is a matrix multiply; serve 512 requests one by one, then in batches
rng = np.random.default_rng(0); W = rng.standard_normal((2048, 2048), dtype=np.float32); X = rng.standard_normal((512, 2048), dtype=np.float32)
for b in (1, 8, 32, 128):
t = time.perf_counter()
for i in range(0, 512, b):
X[i:i + b] @ W
dt = time.perf_counter() - t
print(f"batch size {b:>3}: {512 / dt:8.0f} requests per second")
elif q == "quantize":
rng = np.random.default_rng(1); W = rng.standard_normal((1024, 1024)).astype(np.float32); x = rng.standard_normal(1024).astype(np.float32)
scale = np.abs(W).max() / 127; Wq = np.round(W / scale).astype(np.int8)
exact = W @ x; approx = (Wq.astype(np.float32) @ x) * scale
print(f"fp32 weights {W.nbytes / 1e6:.1f} MB, int8 weights {Wq.nbytes / 1e6:.1f} MB ({W.nbytes // Wq.nbytes}x smaller)")
print(f"average error after int8: {100 * np.abs(exact - approx).mean() / np.abs(exact).mean():.2f}% of the output size")
# One VM for the lab: 2 vCPUs, 8 GB memory, no GPU
gcloud compute instances create ai-vm --zone=us-east1-b --machine-type=e2-standard-2 --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=20GB --service-account=lab-shell@$PROJECT_ID.iam.gserviceaccount.com --scopes=cloud-platform > /dev/null 2>&1 && echo "ai-vm created"
until gcloud compute ssh ai-vm --zone=us-east1-b --tunnel-through-iap --command="true" > /dev/null 2>&1; do sleep 10; done; echo "SSH works"ai-vm created
SSH works# numpy in a Python environment
sudo apt-get update -qq > /dev/null && sudo apt-get install -y -qq python3-venv > /dev/null 2>&1; python3 -m venv .venv && . .venv/bin/activate && pip install -q "numpy==2.*" 2>&1 | grep -v notice | tail -1; python3 -c "import numpy; print('numpy', numpy.__version__)"; nproc; free -g | awk '/Mem/ {print $2 " GB memory"}'numpy 2.4.6
2
7 GB memory2 vCPUs and about 7 GB of usable memory, no GPU.
Q1. How much GPU memory does a model need?
Answer: first, the weights: number of parameters times bytes per parameter (4 for fp32, 2 for fp16 or bf16, 1 for int8, half a byte for int4). Then the KV cache: for every user and every token in the context, each layer keeps keys and values. For chat models with long context, the KV cache can need as much memory as the weights.
Proof:
# Q1. How much GPU memory does a model need? Weights (parameters x bytes) plus the KV cache
. .venv/bin/activate; python3 infra.py memory7B model fp32 28 GB | fp16/bf16 14 GB | int8 7 GB | int4 3.5 GB
13B model fp32 52 GB | fp16/bf16 26 GB | int8 13 GB | int4 6.5 GB
70B model fp32 280 GB | fp16/bf16 140 GB | int8 70 GB | int4 35.0 GB
KV cache for 16 users x 8192 tokens: 17.2 GB more (this is why long context needs big GPUs)A 7B model needs 14 GB in fp16 just for weights, which is why it does not fit a 16 GB GPU together with a big KV cache, and why a 70B model needs several GPUs or 4-bit quantization. The KV cache line uses the shape of a typical 7 to 8B model (32 layers, 8 key-value heads of 128 dimensions): 16 users with 8,192 tokens each add 17 GB more.
Q2. Why do inference servers batch requests?
Answer: a processor does a big matrix multiply much more efficiently than many small ones: it reuses the weights already loaded into fast memory. Serving requests in batches raises throughput, at the cost of some waiting time for each request. GPU servers (vLLM, Triton, TGI) batch continuously.
Proof: the same 512 requests through one 2048 x 2048 layer, in batches of 1, 8, 32 and 128:
# Q2. Why do inference servers batch requests? The same work, sent one by one and in batches
. .venv/bin/activate; python3 infra.py batchingbatch size 1: 1121 requests per second
batch size 8: 2266 requests per second
batch size 32: 4065 requests per second
batch size 128: 6622 requests per secondFrom 1,121 to 6,622 requests per second: about 6 times the throughput from batching alone, on the same CPU. On a GPU the gap is far larger.
Q3. What does quantization buy you, and what does it cost?
Answer: storing weights as int8 instead of fp32 makes them 4 times smaller (less memory, less to read per token, so faster), and adds a small error. Whether the error matters is measured on your task, not assumed.
Proof: one 1024 x 1024 weight matrix, rounded to int8 with one scale factor:
# Q3. What does quantization buy you, and what does it cost? fp32 to int8 on one weight matrix
. .venv/bin/activate; python3 infra.py quantizefp32 weights 4.2 MB, int8 weights 1.0 MB (4x smaller)
average error after int8: 1.11% of the output size4 times smaller, with about 1 percent average error on the output of this one layer. Real quantization methods (per-channel scales, GPTQ, AWQ) keep errors lower across a whole model.

Q4. Why is my container slow when the node's CPU looks fine?
🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโน10,620See the training →Answer: a CPU limit (in Kubernetes resources.limits.cpu, or a cgroup quota) throttles the container to its share of time, even when the node is idle. The app looks "slow" while monitoring shows plenty of free CPU on the machine.
Break it: the same Python loop, first without a limit, then with a quota of 50 percent of one CPU (systemd-run puts it in a cgroup, as a container runtime would):
# Q4. Why is my container slow when CPU looks fine? A CPU limit (here 50 percent of one CPU) throttles it
. .venv/bin/activate
T=$(python3 -c "import time; t = time.time(); sum(i * i for i in range(10_000_000)); print(round(time.time() - t, 2))"); echo "no limit: $T s"
sudo systemd-run --quiet --wait --pipe --property=CPUQuota=50% --uid=$(id -u) --working-directory=$PWD .venv/bin/python3 -c "import time; t = time.time(); sum(i * i for i in range(10_000_000)); print('with CPUQuota=50%:', round(time.time() - t, 2), 's')"no limit: 0.99 s
with CPUQuota=50%: 1.9 sThe same work took twice as long. Fix: raise or remove the CPU limit for latency-sensitive inference (keep the request), and watch the throttling metric (container_cpu_cfs_throttled_periods_total in Kubernetes).
Q5. Where should model files live?
Answer: in object storage (Cloud Storage), versioned, and pulled when a server starts or baked into the image for very fast starts. Measure the real download speed: it decides how fast you can scale out.
Proof: a 500 MB file, uploaded to a test bucket, downloaded to the VM, and copied on the VM's own disk:
# Q5. Where should model files live? Copy a 500 MB "model" from Cloud Storage and from the local disk
B=aimodels-$(echo -n $(gcloud config get-value project 2>/dev/null) | md5sum | cut -c1-8)
gcloud storage buckets create gs://$B --location=us-east1 --uniform-bucket-level-access --public-access-prevention > /dev/null && head -c 500000000 /dev/urandom > model.bin && gcloud storage cp -q model.bin gs://$B/ && rm model.bin
t=$(date +%s.%N); gcloud storage cp -q gs://$B/model.bin model.bin; echo "Cloud Storage -> VM: $(python3 -c "print(round(500 / ($(date +%s.%N) - $t)))") MB/s"
t=$(date +%s.%N); cp model.bin copy.bin && sync; echo "local disk copy: $(python3 -c "print(round(500 / ($(date +%s.%N) - $t)))") MB/s"
gcloud storage rm -r gs://$B --quiet > /dev/null 2>&1; rm -f model.bin copy.bin; echo "test bucket and files removed"...
Average throughput: 281.6MiB/s
Cloud Storage -> VM: 147 MB/s
local disk copy: 61 MB/s
test bucket and files removedCloud Storage delivered the file at 147 MB/s, but the VM's own small standard boot disk copied it at only 61 MB/s: on this VM the disk was the slower part, not the network. That is why model servers use fast local SSDs or load models straight into memory, and why a bigger or SSD persistent disk can speed up start-up more than you might expect.

Clean up (delete everything)
# Clean up: delete the VM
gcloud compute instances delete ai-vm --zone=us-east1-b --quiet 2>&1 | tail -1
echo "lab VMs left: $(gcloud compute instances list --filter='name=ai-vm' --format='value(name)' | wc -l)"Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/ai-vm].
lab VMs left: 0Quick answers to remember
| Question | One-line answer | Proved in |
|---|---|---|
| Model memory | Parameters x bytes, plus the KV cache | Q1 |
| Batching | Bigger matrix multiplies = more throughput | Q2 |
| Quantization | 4x smaller in int8, small measurable error | Q3 |
| Slow container | CPU limits throttle even on idle nodes | Q4 |
| Model files | Object storage; check the disk, not only the network | Q5 |
What next
- Serve a model on Kubernetes: Run Your Own LLM on Kubernetes with Ollama.
- Take a model to production: MLOps for Beginners: From Notebook to Production.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamKServe Hands-On: Serverless Model Serving on Kubernetes
What you build: KServe on GKE serving a scikit-learn model and a small language model on CPU, a broken request, and autoscaling under load. Every step tested.
#ai-infrastructure
CareerByteCode TeamRay on Kubernetes with KubeRay: Distributed Python for AI
What you build: a Ray cluster on GKE with KubeRay, parallel Python tasks, a stuck oversized task and a self-cleaning RayJob. Every step tested.
#ai-infrastructure
CareerByteCode TeamRun Your Own LLM on Kubernetes with Ollama (CPU Only)
What you build: Ollama on a CPU-only GKE node with Llama 3.2 1B and Qwen 2.5 0.5B on a persistent disk, real answers and speed in tokens per second, in your own Google Cloud account. Every step tested
#google-cloud
CareerByteCode TeamMLOps Interview Questions Answered with Labs
What you build: 6 MLOps interview answers proved with real scikit-learn experiments: leakage, reproducibility, imbalance, skew, versioning, latency. Every step tested.
💬 Comments (0)
Top · NewestSign in to comment