Linux Performance Troubleshooting: CPU, Memory and IO
What you build: a slow e2-micro diagnosed for real: a CPU hog, a memory leak and an OOM kill, a disk flat out, found with top, vmstat, pidstat and iostat and fixed with renice, ionice and kill.

✍️ Translate this blog into Hindi, Tamil or Telugu
"The server is slow" is the vaguest ticket in operations, and the most common. In this lab you learn to answer it with facts. On a free Google Cloud VM you start three real troublemakers one at a time (a program that eats the CPU, a program that leaks memory until the kernel kills something, and a backup job that keeps the disk busy), and you find each one with uptime, top, vmstat, pidstat, iostat, free and dmesg, then fix it with renice, kill and ionice. At the end you keep a 60-second health check script for any slow Linux server.
Who it is for: beginners and junior engineers who can use a Linux terminal and want a method instead of guessing. Time: about 1 hour. Cost: free. An e2-micro VM in us-east1 is in Google Cloud's Always Free tier (one per account), and we delete it at the end.
Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. Process ids, times and exact numbers will differ on your machine.
🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโน499Build the projects →Part A: what "slow" means
A server has four main resources. A slow server is almost always one of them running out:

- CPU: work waiting to run. Signs: high load average,
us(user) orsy(system) near 100 invmstat. - Memory (RAM): when it runs out, Linux first drops its file cache, then (if there is swap) moves memory to disk, and finally the OOM killer ends a process. Signs: low
availableinfree,Out of memoryindmesg. - Disk IO: programs waiting for the disk. Signs: high
wa(IO wait) invmstat, high%utiliniostat. - Network: covered in our networking lab (see What next).
Words you will use:
- Load average: the average number of processes running or waiting, over the last 1, 5 and 15 minutes. Compare it with the number of CPUs (
nproc). On 2 CPUs, a load of 2 means "fully busy", 4 means "a queue as long again is waiting". Real-life example: the queue at a bank with 2 counters. - **
vmstatcolumns**:rprocesses wanting CPU,bprocesses blocked on IO,free/buff/cachememory,si/soswap in and out,bi/boblocks read and written, and CPU time split intoususer,sysystem,ididle,waIO wait andststeal (time the cloud host gave to other VMs). - Nice value: a process's politeness, from -20 (greedy) to 19 (very polite).
renicechanges it. ionice does the same for disk access.
The method in one line: look at the whole machine first, then find the process, then fix the cause.
Before you start
- A Google Cloud account (the free trial works) and a project.
- Open Cloud Shell (the
>_icon at the top right of the Google Cloud console) and set your project:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.comPart B: the lab
Step 1: a free e2-micro VM
Why: a small Linux server (Debian 12, 2 shared vCPUs, 1 GB of RAM) where we can cause trouble safely. Small machines show problems quickly, which is good for learning.
gcloud compute instances create perf-lab --zone=us-east1-b --machine-type=e2-micro \
--image-family=debian-12 --image-project=debian-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard \
--no-service-account --no-scopes 2>&1 | tail -2
until gcloud compute ssh perf-lab --zone=us-east1-b --tunnel-through-iap --command="echo ssh ok" 2>/dev/null | grep -q ok; do sleep 10; done; echo "SSH works"NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
perf-lab us-east1-b e2-micro 10.142.0.30 EXTERNAL_IP RUNNING
SSH worksConnect with Compute Engine > VM instances > SSH next to perf-lab, or gcloud compute ssh perf-lab --zone=us-east1-b. Everything from Step 2 runs inside the VM in ~/lab275. Create the file cache-warmer.py there (it is our "leaky" program for the memory test):
# cache-warmer.py: a "leaky" program for the lab. It grabs about 50 MB more memory every second and never gives it back.
import time, sys
limit_mb = int(sys.argv[1]) if len(sys.argv) > 1 else 600
chunks = []
while len(chunks) * 50 < limit_mb:
chunks.append(b'x' * (50 * 1024 * 1024))
time.sleep(1)
time.sleep(600)Step 2: a healthy baseline
Why: you can only spot "abnormal" if you know "normal". sysstat gives iostat, mpstat and pidstat. nproc shows 2 CPUs, uptime the load, free -h the memory, and vmstat 1 3 three one-second samples (ignore the first line: it is the average since boot).
# Install the tools (sysstat = iostat, mpstat, pidstat) and look at a healthy server first
sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq sysstat > /dev/null
nproc; uptime
free -h
vmstat 1 32
20:45:27 up 1 min, 0 user, load average: 0.43, 0.15, 0.05
total used free shared buff/cache available
Mem: 969Mi 358Mi 164Mi 472Ki 587Mi 611Mi
Swap: 0B 0B 0B
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 168748 25664 575840 0 0 3276 929 442 557 9 6 76 9 0
1 0 0 168748 25664 575840 0 0 0 0 268 379 1 1 99 0 0
1 0 0 168748 25664 575840 0 0 0 0 57 55 0 0 100 0 0
This VM has no swap (Swap: 0B), like most cloud images. That matters for the memory test: with nothing to swap to, running out of memory goes straight to the OOM killer.
Break it: something eats the CPU
We start two copies of a "mystery" program. Each runs sha256sum /dev/zero, which hashes an endless stream of zeros and uses a whole CPU. exec -a report-builder gives it a harmless-looking name, as real runaway jobs often have.
# Break it: a "mystery" program eats the CPU. Start it, then find it like an on-call engineer.
for i in 1 2; do nohup bash -c 'exec -a report-builder sha256sum /dev/zero' > /dev/null 2>&1 & done
sleep 20
uptime
top -b -n 1 -o %CPU | sed -n '1,3p;7,10p'
vmstat 1 5 20:45:59 up 1 min, 0 user, load average: 0.83, 0.27, 0.10
top - 20:45:59 up 1 min, 0 user, load average: 0.83, 0.27, 0.10
Tasks: 98 total, 3 running, 95 sleeping, 0 stopped, 0 zombie
%Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1726 you 20 0 2932 896 808 R 95.8 0.1 0:20.82 sha256s+
1727 you 20 0 2932 912 824 R 95.8 0.1 0:20.63 sha256s+
1 root 20 0 168156 12528 9204 S 0.0 1.3 0:02.92 systemd
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 0 176856 25712 576252 0 0 2310 857 370 411 25 5 64 6 0
3 0 0 176856 25712 576252 0 0 0 0 88 54 97 3 0 0 0
3 0 0 176856 25712 576252 0 0 0 0 80 25 100 0 0 0 0
3 0 0 176856 25712 576252 0 0 0 0 76 18 97 3 0 0 0
3 0 0 176856 25712 576252 0 0 0 72 82 39 100 0 0 0 0The machine-wide view says CPU: %Cpu(s): ... 0.0 id (no idle time left) and vmstat shows r = 3 (three processes want CPU on 2 CPUs) with us at 96 to 100. top sorted by CPU shows two processes at 100 percent each. Note that top shows the program (sha256sum, cut to sha256s+), not the name it pretends to be.
Find the CPU hog
Why: pidstat -u 1 3 measures CPU per process over 3 seconds (more exact than one top snapshot), and ps with --sort=-%cpu shows the full command line, the owner and how long it has been running (ELAPSED).
# Who exactly? pidstat per process, and ps sorted by CPU
pidstat -u 1 3 | grep -E "Average|report" | head -4
ps -eo pid,user,ni,%cpu,etime,cmd --sort=-%cpu | grep -v "ps -eo" | head -3Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 1000 1726 99.34 0.66 0.00 0.00 100.00 - sha256sum
Average: 1000 1727 98.34 0.66 0.00 1.33 99.00 - sha256sum
Average: 1000 1774 0.00 0.66 0.00 0.00 0.66 - pidstat
PID USER NI %CPU ELAPSED CMD
1726 you 0 90.3 00:48 report-builder /dev/zero
1727 you 0 89.8 00:48 report-builder /dev/zero
Now we know: two processes, owned by our user, named report-builder, each using about one full CPU, running for under a minute.
Fix it: renice, then kill
Why: if a job must finish, first make it polite: renice -n 19 puts it at the back of the queue, so everything else runs first. If it should not run at all, stop it: pkill -f kills processes whose command line matches.
# Fix it: first lower its priority (renice), then stop it (kill), then check
for p in $(pgrep -f report-builder); do sudo renice -n 19 -p $p; done
pkill -f report-builder; sleep 15
pgrep -f report-builder || echo "report-builder stopped"
uptime; vmstat 1 2 | tail -11726 (process ID) old priority 0, new priority 19
1727 (process ID) old priority 0, new priority 19
report-builder stopped
20:47:02 up 2 min, 0 user, load average: 1.50, 0.61, 0.23
1 0 0 183624 25780 576752 0 0 0 0 37 35 0 0 100 0 0The CPU is idle again (id 100). The load average is still above 1 because it is an average over the last minute; it falls over the next minutes. Do not panic about a load number that is already dropping.
Break it: a memory leak
cache-warmer.py grabs 50 MB more every second up to 600 MB and never lets go, like a real app with a memory leak.
# Break it: a program that keeps eating memory (a memory leak)
nohup python3 cache-warmer.py 600 > /dev/null 2>&1 &
sleep 15
free -h
ps -eo pid,%mem,rss,cmd --sort=-rss | head -3
sleep 8; ps -o rss= -p $(pgrep -o -f "^python3 cache-warmer") | awk '{print "cache-warmer RSS now", int($1/1024), "MB"}' total used free shared buff/cache available
Mem: 969Mi 924Mi 64Mi 468Ki 101Mi 45Mi
Swap: 0B 0B 0B
PID %MEM RSS CMD
1867 62.3 619400 python3 cache-warmer.py 600
451 2.1 21304 /usr/bin/google_osconfig_agent
cache-warmer RSS now 604 MBfree -h shows available down to tens of MB on a 1 GB machine, and the file cache (buff/cache) squeezed out to make room. ps --sort=-rss (RSS = memory really in use) points at the leaking program at about 600 MB, over 60 percent of all memory.
The OOM killer
Now a second program asks for 1.5 GB, more than the whole VM has. Linux has no swap to fall back on, so the kernel's out-of-memory (OOM) killer picks a process and ends it.
# Break it harder: ask for more memory than the VM has. The kernel's OOM killer steps in.
python3 -c "x = b'x' * (1500 * 1024 * 1024)"; echo "exit code $? (137 = killed by signal 9)"
sudo dmesg | grep -iE "out of memory|oom-kill" | tail -2 | sed 's/^\[[^]]*\] //'bash: 1920 Killed python3 -c "x = b'x' * (1500 * 1024 * 1024)"
exit code 137 (137 = killed by signal 9)
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-8.scope,task=python3,pid=1920,uid=1000
Out of memory: Killed process 1920 (python3) total-vm:1549860kB, anon-rss:725244kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:1484kB oom_score_adj:0
The shell reports Killed and exit code 137 (128 + signal 9, SIGKILL). dmesg (the kernel's own log) tells you exactly what happened: Out of memory: Killed process ... (python3) with the memory it held. **Whenever a program dies with exit code 137 or "Killed" and no error of its own, check dmesg for the OOM killer first.** The same happens in containers and Kubernetes, where it shows as OOMKilled.
Fix it: stop the leak
# Fix it: stop the leaking program and check the memory comes back
pkill -f cache-warmer; sleep 3
free -h | head -2 total used free shared buff/cache available
Mem: 969Mi 331Mi 629Mi 468Ki 142Mi 638MiMemory is back. The real fix for a leak is in the code (or a restart schedule while it is fixed), plus a memory limit so one program cannot take the whole machine (for example MemoryMax= in a systemd unit or a container memory limit).
Break it: the disk is busy
A "backup job" writes 1 GB files to the disk again and again, with oflag=direct so the writes really go to the disk instead of the memory cache.
# Break it: a backup job that writes the disk flat out
nohup bash -c 'exec -a backup-job bash -c "while :; do dd if=/dev/zero of=$HOME/lab275/big.img bs=1M count=1000 oflag=direct status=none; done"' > /dev/null 2>&1 &
sleep 10
vmstat 1 4
iostat -dx 2 2 | awk '/^Device/ {h=$0} /^sda/ {l=$0} END {print h; print l}' | awk '{print $1, $2, $8, $9, $NF}' | column -tprocs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 0 599952 14764 168516 0 0 1946 2804 273 305 16 4 72 8 0
1 1 0 599952 14764 168516 0 0 0 104456 771 320 1 3 49 48 1
1 1 0 599952 14764 168516 0 0 0 104448 740 285 0 2 50 48 0
1 1 0 599952 14764 168516 0 0 0 104448 670 286 0 3 50 48 0
Device r/s w/s wkB/s %util
sda 4.00 415.50 104288.00 94.80vmstat now shows b = 1 (one process blocked waiting for the disk), bo around 100,000 KB written per second, and wa (IO wait) near 50 percent: half the CPU time is spent waiting for the disk. iostat -dx confirms it from the disk's side: sda writes about 100 MB/s and is busy over 90 percent of the time (%util).
Find the IO hog
Why: pidstat -d shows disk reads and writes per process, and ps -C dd shows who started it (PPID is the parent process).
# Who is writing? pidstat -d shows disk use per process
pidstat -d 2 2 | grep -E "Average.*(UID| dd$)" | head -3
ps -o pid,ppid,cmd -C ddAverage: UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
Average: 1000 2015 0.00 104187.53 0.00 0 dd
PID PPID CMD
2015 1997 dd if=/dev/zero of=/home/you/lab275/big.img bs=1M count=1000 oflag=direct status=noneFix it: ionice, or stop it
Why: ionice -c 3 puts a process in the idle IO class: it only gets the disk when nobody else needs it. Good for backups that must run during the day. Or stop it.
# Fix it: run it with idle IO priority (ionice), or stop it
for p in $(pgrep -f backup-job) $(pgrep -x dd); do sudo ionice -c 3 -p $p && echo "ionice idle for $p"; done
ionice -p $(pgrep -x dd | head -1)
pkill -f backup-job; pkill -x dd; sleep 2; rm -f big.img
vmstat 1 2 | tail -1ionice idle for 1997
ionice idle for 2052
idle
1 0 0 597088 15320 171656 0 0 0 0 50 48 0 0 100 0 0
Step 3: a 60-second health check
Why: when you log in to a slow server, run the same checks in the same order every time. This small script does the first minute for you: load, the last kernel messages, CPU/memory/IO, memory, disk space, and the top 3 CPU users.
# The 60-second check: one script you can run on any slow Linux server
cat > health.sh <<'SH'
#!/bin/bash
echo "== load"; uptime
echo "== kernel"; sudo dmesg | tail -2 | sed 's/^\[[^]]*\] //'
echo "== cpu+mem+io (vmstat)"; vmstat 1 3 | tail -2
echo "== memory"; free -m | head -2
echo "== disk"; df -h / | tail -1
echo "== top 3 cpu"; ps -eo pid,%cpu,comm --sort=-%cpu | sed -n 2,4p
SH
chmod +x health.sh && ./health.sh== load
20:49:13 up 5 min, 0 user, load average: 0.69, 0.58, 0.27
== kernel
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-8.scope,task=python3,pid=1920,uid=1000
Out of memory: Killed process 1920 (python3) total-vm:1549860kB, anon-rss:725244kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:1484kB oom_score_adj:0
== cpu+mem+io (vmstat)
1 0 0 598472 15340 171844 0 0 0 0 42 43 0 0 100 0 0
1 0 0 598472 15340 171844 0 0 0 0 41 41 0 0 100 0 0
== memory
total used free shared buff/cache available
Mem: 969 336 584 0 182 633
== disk
/dev/sda1 9.7G 2.1G 7.1G 23% /
== top 3 cpu
1 0.9 systemd
207 0.5 systemd-journal
2104 0.5 sshd
Notice that the kernel section still shows the OOM kill from earlier. dmesg keeps history, which is exactly why it belongs in a health check.
POC: prove it works
- Two runaway processes pushed CPU idle to 0 and the run queue to 3;
pidstatandpsnamed them;reniceandkillbrought idle back to 100. - A leaking program took about 600 MB of 1 GB; a 1.5 GB request was killed by the OOM killer (exit 137) and
dmesgexplained it. - A write-heavy job pushed IO wait near 50 percent and disk use over 90 percent;
pidstat -dfounddd;ioniceandkillfixed it. - A reusable 60-second health check script.
Quiz task: check what you learned
- A 4-CPU server has a load average of 3.5. Is it overloaded? (Answer: no, it is busy but under 4. At 8 or more, work is queueing.)
vmstatshowswaat 60. What do you check next? *(Answer: the disk:iostat -dx 1andpidstat -d 1to find the process.)*- A program dies with exit code 137 and no error message. First command? *(Answer:
sudo dmesg | grep -i "out of memory".)* - A nightly report must still run but slows the web app. What do you do? *(Answer: run it with
nice -n 19andionice -c 3, or move it to a quiet hour.)* - Which
freecolumn tells you how much memory is really left for new programs? *(Answer:available.)*
Clean up (delete everything)
Exit the VM (exit), then in Cloud Shell:
gcloud compute instances delete perf-lab --zone=us-east1-b --quiet 2>&1 | tail -1
echo "perf-lab VMs left: $(gcloud compute instances list --filter="name=perf-lab" --format="value(name)" | wc -l)"
echo "perf-lab disks left: $(gcloud compute disks list --filter="name~^perf-lab" --format="value(name)" | wc -l)"Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/perf-lab].
WARNING: The following filter keys were not present in any resource : name
perf-lab VMs left: 0
WARNING: The following filter keys were not present in any resource : name
perf-lab disks left: 0
FAQ
**top or htop?** Both show the same data. htop is friendlier (colours, scrolling, tree view) but often not installed; top is everywhere. top -c shows full command lines.
Why is "free" memory so low on a healthy server? Linux uses spare memory as file cache to make things faster. That memory is given back at once when programs need it, which is why available matters, not free.
**What is steal time (st)?** Time your VM wanted the CPU but the cloud host was running another VM. Shared-core machines like e2-micro can show it under load. High steal all the time means you need a bigger or non-shared machine type.
Should I add swap to a cloud VM? A small swap file can stop sudden OOM kills on tiny VMs, but it makes the server slow instead of failing fast. For production, size memory properly and set memory limits.
How do I watch this over time instead of in the moment? sar (from sysstat) records history once it is enabled, and Cloud Monitoring with the Ops Agent graphs CPU, memory and disk for every VM (see What next).
Can these commands harm the server? The read-only ones (uptime, top, vmstat, iostat, pidstat, free, dmesg) are safe anywhere. Be careful with kill on production: make sure you know what the process is first.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
| Load average above the number of CPUs | CPU queue or processes stuck on IO | vmstat 1: high us/sy = CPU, high b/wa = disk |
A process dies with exit 137 or Killed | The OOM killer ended it | sudo dmesg; fix the leak, add memory or a limit |
iostat: command not found | sysstat is not installed | sudo apt install sysstat |
High wa, but pidstat -d shows nothing | The process ended, or writes are buffered | Run pidstat -d 1 while it happens; check iotop |
ionice: option requires an argument -- 'p' | pgrep found no process, so no PID was passed | The process already ended; check pgrep -a <name> |
High st (steal) | The shared host is busy | Use a non-shared machine type for steady load |
What next
- Turn these checks into dashboards and alerts: Monitor Linux Servers with Node Exporter and Ops Agent.
- The fourth resource, the network: Linux for Cloud Beginners: 50 Commands on a Free VM.
- Find errors in logs fast: Cloud Logging Hands-On: Find Any Error in Seconds.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamLinux Disk and Storage: Mount, Resize and Fix a Full Disk
What you build: a data disk added, mounted and kept in fstab, filled to 100 percent, a deleted-but-open file found, the disk resized live from 10 to 20 GB, and an inode-full disk fixed.
#devops
CareerByteCode TeamSelf-Host a Git Server and CI Runner on a Free VM
What you build: Gitea and its CI runner (act_runner) on one free e2-micro VM; a push runs the tests, a bug turns CI red with the failing test shown, a revert turns it green, and gitea dump backs it al
#beginners
CareerByteCode TeamCron Jobs and Log Rotation on Linux
What you build: a disk report job run by cron every minute with its log rotated by logrotate, on a free VM, plus 3 real mistakes: relative paths, the % sign and an insecure log folder.
#beginners
CareerByteCode TeamLinux Users, Groups and sudo Explained
What you build: a shared Linux server for 3 people on a free VM: users, a developers group, a sudo rule that allows one job only, visudo catching a typo, the sudo log, and an account locked and remove
💬 Comments (0)
Top · NewestSign in to comment