CareerByteCode
ByteBlogby CareerByteCode

AI Infrastructure Engineer Interview Questions with Labs

What you build: 5 AI infrastructure interview answers proved on a CPU VM: model memory, batching, quantization, CPU throttling and model storage speed. Every step tested.

AI Infrastructure Engineer Interview Questions with Labs

✍️ Translate this blog into Hindi, Tamil or Telugu

AI infrastructure interviews ask engineering questions, not model trivia: how much memory does a model need, why do inference servers batch, what does quantization buy, why is my container slow, where should model files live? You can answer them with numbers you measured yourself, on a CPU, in minutes. In this post, every answer is proved with a small, real experiment on a Google Cloud VM (2 vCPUs, 8 GB memory, no GPU).

Who it is for: engineers moving into AI platform, MLOps or ML infrastructure roles. Time: about 45 minutes. Cost: one small VM and a temporary bucket for about 10 minutes, covered by the free trial credit. No GPU or TPU is used.

Every command below was run for real by the CareerByteCode Team, and every output is the real result we got. The VM was created with gcloud from a Cloud Shell-like Linux machine; the experiments ran on the VM over SSH.

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →

The lab behind the answers

All experiments are in one small Python file that uses only numpy (a fast library for arrays and matrix maths). A neural network layer is, at its core, a matrix multiply, so numpy is enough to show how the hardware behaves. Save it as infra.py:

# infra.py <question>: small, real measurements behind common AI infrastructure interview answers (CPU only, numpy).
import sys, time
import numpy as np

q = sys.argv[1]

if q == "memory":
    # weights only: parameters x bytes per parameter; plus the KV cache a chat model keeps per token
    for name, params in (("7B model", 7e9), ("13B model", 13e9), ("70B model", 70e9)):
        print(f"{name:<10} fp32 {params * 4 / 1e9:6.0f} GB | fp16/bf16 {params * 2 / 1e9:5.0f} GB | int8 {params / 1e9:5.0f} GB | int4 {params / 2 / 1e9:5.1f} GB")
    layers, kv_heads, head_dim, tokens, users = 32, 8, 128, 8192, 16      # a typical 7-8B model with grouped-query attention
    kv = 2 * layers * kv_heads * head_dim * 2 * tokens * users              # keys + values, fp16
    print(f"KV cache for {users} users x {tokens} tokens: {kv / 1e9:.1f} GB more (this is why long context needs big GPUs)")

elif q == "batching":
    # one "layer" of a model is a matrix multiply; serve 512 requests one by one, then in batches
    rng = np.random.default_rng(0); W = rng.standard_normal((2048, 2048), dtype=np.float32); X = rng.standard_normal((512, 2048), dtype=np.float32)
    for b in (1, 8, 32, 128):
        t = time.perf_counter()
        for i in range(0, 512, b):
            X[i:i + b] @ W
        dt = time.perf_counter() - t
        print(f"batch size {b:>3}: {512 / dt:8.0f} requests per second")

elif q == "quantize":
    rng = np.random.default_rng(1); W = rng.standard_normal((1024, 1024)).astype(np.float32); x = rng.standard_normal(1024).astype(np.float32)
    scale = np.abs(W).max() / 127; Wq = np.round(W / scale).astype(np.int8)
    exact = W @ x; approx = (Wq.astype(np.float32) @ x) * scale
    print(f"fp32 weights {W.nbytes / 1e6:.1f} MB, int8 weights {Wq.nbytes / 1e6:.1f} MB ({W.nbytes // Wq.nbytes}x smaller)")
    print(f"average error after int8: {100 * np.abs(exact - approx).mean() / np.abs(exact).mean():.2f}% of the output size")
The lab
One VM, numpy experiments for memory, batching and quantization, a CPU limit test and a Cloud Storage download test
# One VM for the lab: 2 vCPUs, 8 GB memory, no GPU
gcloud compute instances create ai-vm --zone=us-east1-b --machine-type=e2-standard-2 --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=20GB --service-account=lab-shell@$PROJECT_ID.iam.gserviceaccount.com --scopes=cloud-platform > /dev/null 2>&1 && echo "ai-vm created"
until gcloud compute ssh ai-vm --zone=us-east1-b --tunnel-through-iap --command="true" > /dev/null 2>&1; do sleep 10; done; echo "SSH works"
ai-vm created
SSH works
# numpy in a Python environment
sudo apt-get update -qq > /dev/null && sudo apt-get install -y -qq python3-venv > /dev/null 2>&1; python3 -m venv .venv && . .venv/bin/activate && pip install -q "numpy==2.*" 2>&1 | grep -v notice | tail -1; python3 -c "import numpy; print('numpy', numpy.__version__)"; nproc; free -g | awk '/Mem/ {print $2 " GB memory"}'
numpy 2.4.6
2
7 GB memory

2 vCPUs and about 7 GB of usable memory, no GPU.

Q1. How much GPU memory does a model need?

Answer: first, the weights: number of parameters times bytes per parameter (4 for fp32, 2 for fp16 or bf16, 1 for int8, half a byte for int4). Then the KV cache: for every user and every token in the context, each layer keeps keys and values. For chat models with long context, the KV cache can need as much memory as the weights.

Proof:

# Q1. How much GPU memory does a model need? Weights (parameters x bytes) plus the KV cache
. .venv/bin/activate; python3 infra.py memory
7B model   fp32     28 GB | fp16/bf16    14 GB | int8     7 GB | int4   3.5 GB
13B model  fp32     52 GB | fp16/bf16    26 GB | int8    13 GB | int4   6.5 GB
70B model  fp32    280 GB | fp16/bf16   140 GB | int8    70 GB | int4  35.0 GB
KV cache for 16 users x 8192 tokens: 17.2 GB more (this is why long context needs big GPUs)

A 7B model needs 14 GB in fp16 just for weights, which is why it does not fit a 16 GB GPU together with a big KV cache, and why a 70B model needs several GPUs or 4-bit quantization. The KV cache line uses the shape of a typical 7 to 8B model (32 layers, 8 key-value heads of 128 dimensions): 16 users with 8,192 tokens each add 17 GB more.

Q2. Why do inference servers batch requests?

Answer: a processor does a big matrix multiply much more efficiently than many small ones: it reuses the weights already loaded into fast memory. Serving requests in batches raises throughput, at the cost of some waiting time for each request. GPU servers (vLLM, Triton, TGI) batch continuously.

Proof: the same 512 requests through one 2048 x 2048 layer, in batches of 1, 8, 32 and 128:

# Q2. Why do inference servers batch requests? The same work, sent one by one and in batches
. .venv/bin/activate; python3 infra.py batching
batch size   1:     1121 requests per second
batch size   8:     2266 requests per second
batch size  32:     4065 requests per second
batch size 128:     6622 requests per second

From 1,121 to 6,622 requests per second: about 6 times the throughput from batching alone, on the same CPU. On a GPU the gap is far larger.

Q3. What does quantization buy you, and what does it cost?

Answer: storing weights as int8 instead of fp32 makes them 4 times smaller (less memory, less to read per token, so faster), and adds a small error. Whether the error matters is measured on your task, not assumed.

Proof: one 1024 x 1024 weight matrix, rounded to int8 with one scale factor:

# Q3. What does quantization buy you, and what does it cost? fp32 to int8 on one weight matrix
. .venv/bin/activate; python3 infra.py quantize
fp32 weights 4.2 MB, int8 weights 1.0 MB (4x smaller)
average error after int8: 1.11% of the output size

4 times smaller, with about 1 percent average error on the output of this one layer. Real quantization methods (per-channel scales, GPTQ, AWQ) keep errors lower across a whole model.

Memory, batching, quantization
Weights and KV cache per model size; 1,121 to 6,622 requests a second with batching; int8 is 4x smaller with about 1 percent error

Q4. Why is my container slow when the node's CPU looks fine?

🎓⭐ Recommended by CareerByteCode · Live trainingLearn AI, Build 12+ AI Agentsโ‚น10,620See the training →

Answer: a CPU limit (in Kubernetes resources.limits.cpu, or a cgroup quota) throttles the container to its share of time, even when the node is idle. The app looks "slow" while monitoring shows plenty of free CPU on the machine.

Break it: the same Python loop, first without a limit, then with a quota of 50 percent of one CPU (systemd-run puts it in a cgroup, as a container runtime would):

# Q4. Why is my container slow when CPU looks fine? A CPU limit (here 50 percent of one CPU) throttles it
. .venv/bin/activate
T=$(python3 -c "import time; t = time.time(); sum(i * i for i in range(10_000_000)); print(round(time.time() - t, 2))"); echo "no limit: $T s"
sudo systemd-run --quiet --wait --pipe --property=CPUQuota=50% --uid=$(id -u) --working-directory=$PWD .venv/bin/python3 -c "import time; t = time.time(); sum(i * i for i in range(10_000_000)); print('with CPUQuota=50%:', round(time.time() - t, 2), 's')"
no limit: 0.99 s
with CPUQuota=50%: 1.9 s

The same work took twice as long. Fix: raise or remove the CPU limit for latency-sensitive inference (keep the request), and watch the throttling metric (container_cpu_cfs_throttled_periods_total in Kubernetes).

Q5. Where should model files live?

Answer: in object storage (Cloud Storage), versioned, and pulled when a server starts or baked into the image for very fast starts. Measure the real download speed: it decides how fast you can scale out.

Proof: a 500 MB file, uploaded to a test bucket, downloaded to the VM, and copied on the VM's own disk:

# Q5. Where should model files live? Copy a 500 MB "model" from Cloud Storage and from the local disk
B=aimodels-$(echo -n $(gcloud config get-value project 2>/dev/null) | md5sum | cut -c1-8)
gcloud storage buckets create gs://$B --location=us-east1 --uniform-bucket-level-access --public-access-prevention > /dev/null && head -c 500000000 /dev/urandom > model.bin && gcloud storage cp -q model.bin gs://$B/ && rm model.bin
t=$(date +%s.%N); gcloud storage cp -q gs://$B/model.bin model.bin; echo "Cloud Storage -> VM: $(python3 -c "print(round(500 / ($(date +%s.%N) - $t)))") MB/s"
t=$(date +%s.%N); cp model.bin copy.bin && sync; echo "local disk copy: $(python3 -c "print(round(500 / ($(date +%s.%N) - $t)))") MB/s"
gcloud storage rm -r gs://$B --quiet > /dev/null 2>&1; rm -f model.bin copy.bin; echo "test bucket and files removed"
...

Average throughput: 281.6MiB/s
Cloud Storage -> VM: 147 MB/s
local disk copy: 61 MB/s
test bucket and files removed

Cloud Storage delivered the file at 147 MB/s, but the VM's own small standard boot disk copied it at only 61 MB/s: on this VM the disk was the slower part, not the network. That is why model servers use fast local SSDs or load models straight into memory, and why a bigger or SSD persistent disk can speed up start-up more than you might expect.

Throttling and storage
A 50 percent CPU quota doubles the time; Cloud Storage 147 MB/s to the VM, the standard disk 61 MB/s

Clean up (delete everything)

# Clean up: delete the VM
gcloud compute instances delete ai-vm --zone=us-east1-b --quiet 2>&1 | tail -1
echo "lab VMs left: $(gcloud compute instances list --filter='name=ai-vm' --format='value(name)' | wc -l)"
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/ai-vm].
lab VMs left: 0

Quick answers to remember

QuestionOne-line answerProved in
Model memoryParameters x bytes, plus the KV cacheQ1
BatchingBigger matrix multiplies = more throughputQ2
Quantization4x smaller in int8, small measurable errorQ3
Slow containerCPU limits throttle even on idle nodesQ4
Model filesObject storage; check the disk, not only the networkQ5

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →KServe Hands-On: Serverless Model Serving on Kubernetes⏱ 11 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🎓⭐ Recommended by CareerByteCode · Live trainingAI Engineering and Agentic AI Programโ‚น35,400See the training →