CareerByteCode
ByteBlogby CareerByteCode

Linux Performance Troubleshooting: CPU, Memory and IO

What you build: a slow e2-micro diagnosed for real: a CPU hog, a memory leak and an OOM kill, a disk flat out, found with top, vmstat, pidstat and iostat and fixed with renice, ionice and kill.

Linux Performance Troubleshooting: CPU, Memory and IO

✍️ Translate this blog into Hindi, Tamil or Telugu

"The server is slow" is the vaguest ticket in operations, and the most common. In this lab you learn to answer it with facts. On a free Google Cloud VM you start three real troublemakers one at a time (a program that eats the CPU, a program that leaks memory until the kernel kills something, and a backup job that keeps the disk busy), and you find each one with uptime, top, vmstat, pidstat, iostat, free and dmesg, then fix it with renice, kill and ionice. At the end you keep a 60-second health check script for any slow Linux server.

Who it is for: beginners and junior engineers who can use a Linux terminal and want a method instead of guessing. Time: about 1 hour. Cost: free. An e2-micro VM in us-east1 is in Google Cloud's Always Free tier (one per account), and we delete it at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. Process ids, times and exact numbers will differ on your machine.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →

Part A: what "slow" means

A server has four main resources. A slow server is almost always one of them running out:

Performance in one picture
Check load first, then CPU, memory and disk, find the process with pidstat or ps, then fix it
  • CPU: work waiting to run. Signs: high load average, us (user) or sy (system) near 100 in vmstat.
  • Memory (RAM): when it runs out, Linux first drops its file cache, then (if there is swap) moves memory to disk, and finally the OOM killer ends a process. Signs: low available in free, Out of memory in dmesg.
  • Disk IO: programs waiting for the disk. Signs: high wa (IO wait) in vmstat, high %util in iostat.
  • Network: covered in our networking lab (see What next).

Words you will use:

  • Load average: the average number of processes running or waiting, over the last 1, 5 and 15 minutes. Compare it with the number of CPUs (nproc). On 2 CPUs, a load of 2 means "fully busy", 4 means "a queue as long again is waiting". Real-life example: the queue at a bank with 2 counters.
  • **vmstat columns**: r processes wanting CPU, b processes blocked on IO, free/buff/cache memory, si/so swap in and out, bi/bo blocks read and written, and CPU time split into us user, sy system, id idle, wa IO wait and st steal (time the cloud host gave to other VMs).
  • Nice value: a process's politeness, from -20 (greedy) to 19 (very polite). renice changes it. ionice does the same for disk access.

The method in one line: look at the whole machine first, then find the process, then fix the cause.

Before you start

  1. A Google Cloud account (the free trial works) and a project.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console) and set your project:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.com

Part B: the lab

Step 1: a free e2-micro VM

Why: a small Linux server (Debian 12, 2 shared vCPUs, 1 GB of RAM) where we can cause trouble safely. Small machines show problems quickly, which is good for learning.

gcloud compute instances create perf-lab --zone=us-east1-b --machine-type=e2-micro \
  --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard \
  --no-service-account --no-scopes 2>&1 | tail -2
until gcloud compute ssh perf-lab --zone=us-east1-b --tunnel-through-iap --command="echo ssh ok" 2>/dev/null | grep -q ok; do sleep 10; done; echo "SSH works"
NAME      ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
perf-lab  us-east1-b  e2-micro                   10.142.0.30  EXTERNAL_IP  RUNNING
SSH works

Connect with Compute Engine > VM instances > SSH next to perf-lab, or gcloud compute ssh perf-lab --zone=us-east1-b. Everything from Step 2 runs inside the VM in ~/lab275. Create the file cache-warmer.py there (it is our "leaky" program for the memory test):

# cache-warmer.py: a "leaky" program for the lab. It grabs about 50 MB more memory every second and never gives it back.
import time, sys
limit_mb = int(sys.argv[1]) if len(sys.argv) > 1 else 600
chunks = []
while len(chunks) * 50 < limit_mb:
    chunks.append(b'x' * (50 * 1024 * 1024))
    time.sleep(1)
time.sleep(600)

Step 2: a healthy baseline

Why: you can only spot "abnormal" if you know "normal". sysstat gives iostat, mpstat and pidstat. nproc shows 2 CPUs, uptime the load, free -h the memory, and vmstat 1 3 three one-second samples (ignore the first line: it is the average since boot).

# Install the tools (sysstat = iostat, mpstat, pidstat) and look at a healthy server first
sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq sysstat > /dev/null
nproc; uptime
free -h
vmstat 1 3
2
 20:45:27 up 1 min,  0 user,  load average: 0.43, 0.15, 0.05
               total        used        free      shared  buff/cache   available
Mem:           969Mi       358Mi       164Mi       472Ki       587Mi       611Mi
Swap:             0B          0B          0B
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 168748  25664 575840    0    0  3276   929  442  557  9  6 76  9  0
 1  0      0 168748  25664 575840    0    0     0     0  268  379  1  1 99  0  0
 1  0      0 168748  25664 575840    0    0     0     0   57   55  0  0 100  0  0
Baseline
2 CPUs, about 600 MB of memory available, no swap, and CPU 99 percent idle after the first line

This VM has no swap (Swap: 0B), like most cloud images. That matters for the memory test: with nothing to swap to, running out of memory goes straight to the OOM killer.

Break it: something eats the CPU

We start two copies of a "mystery" program. Each runs sha256sum /dev/zero, which hashes an endless stream of zeros and uses a whole CPU. exec -a report-builder gives it a harmless-looking name, as real runaway jobs often have.

# Break it: a "mystery" program eats the CPU. Start it, then find it like an on-call engineer.
for i in 1 2; do nohup bash -c 'exec -a report-builder sha256sum /dev/zero' > /dev/null 2>&1 & done
sleep 20
uptime
top -b -n 1 -o %CPU | sed -n '1,3p;7,10p'
vmstat 1 5
 20:45:59 up 1 min,  0 user,  load average: 0.83, 0.27, 0.10
top - 20:45:59 up 1 min,  0 user,  load average: 0.83, 0.27, 0.10
Tasks:  98 total,   3 running,  95 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1726 you  20   0    2932    896    808 R  95.8   0.1   0:20.82 sha256s+
   1727 you  20   0    2932    912    824 R  95.8   0.1   0:20.63 sha256s+
      1 root      20   0  168156  12528   9204 S   0.0   1.3   0:02.92 systemd
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 176856  25712 576252    0    0  2310   857  370  411 25  5 64  6  0
 3  0      0 176856  25712 576252    0    0     0     0   88   54 97  3  0  0  0
 3  0      0 176856  25712 576252    0    0     0     0   80   25 100  0  0  0  0
 3  0      0 176856  25712 576252    0    0     0     0   76   18 97  3  0  0  0
 3  0      0 176856  25712 576252    0    0     0    72   82   39 100  0  0  0  0

The machine-wide view says CPU: %Cpu(s): ... 0.0 id (no idle time left) and vmstat shows r = 3 (three processes want CPU on 2 CPUs) with us at 96 to 100. top sorted by CPU shows two processes at 100 percent each. Note that top shows the program (sha256sum, cut to sha256s+), not the name it pretends to be.

Find the CPU hog

Why: pidstat -u 1 3 measures CPU per process over 3 seconds (more exact than one top snapshot), and ps with --sort=-%cpu shows the full command line, the owner and how long it has been running (ELAPSED).

# Who exactly? pidstat per process, and ps sorted by CPU
pidstat -u 1 3 | grep -E "Average|report" | head -4
ps -eo pid,user,ni,%cpu,etime,cmd --sort=-%cpu | grep -v "ps -eo" | head -3
Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
Average:     1000      1726   99.34    0.66    0.00    0.00  100.00     -  sha256sum
Average:     1000      1727   98.34    0.66    0.00    1.33   99.00     -  sha256sum
Average:     1000      1774    0.00    0.66    0.00    0.00    0.66     -  pidstat
    PID USER      NI %CPU     ELAPSED CMD
   1726 you   0 90.3       00:48 report-builder /dev/zero
   1727 you   0 89.8       00:48 report-builder /dev/zero
CPU hog found
Load rising, no idle CPU, r=3, and pidstat and ps point at two report-builder processes using a full CPU each

Now we know: two processes, owned by our user, named report-builder, each using about one full CPU, running for under a minute.

Fix it: renice, then kill

Why: if a job must finish, first make it polite: renice -n 19 puts it at the back of the queue, so everything else runs first. If it should not run at all, stop it: pkill -f kills processes whose command line matches.

# Fix it: first lower its priority (renice), then stop it (kill), then check
for p in $(pgrep -f report-builder); do sudo renice -n 19 -p $p; done
pkill -f report-builder; sleep 15
pgrep -f report-builder || echo "report-builder stopped"
uptime; vmstat 1 2 | tail -1
1726 (process ID) old priority 0, new priority 19
1727 (process ID) old priority 0, new priority 19
report-builder stopped
 20:47:02 up 2 min,  0 user,  load average: 1.50, 0.61, 0.23
 1  0      0 183624  25780 576752    0    0     0     0   37   35  0  0 100  0  0

The CPU is idle again (id 100). The load average is still above 1 because it is an average over the last minute; it falls over the next minutes. Do not panic about a load number that is already dropping.

Break it: a memory leak

cache-warmer.py grabs 50 MB more every second up to 600 MB and never lets go, like a real app with a memory leak.

# Break it: a program that keeps eating memory (a memory leak)
nohup python3 cache-warmer.py 600 > /dev/null 2>&1 &
sleep 15
free -h
ps -eo pid,%mem,rss,cmd --sort=-rss | head -3
sleep 8; ps -o rss= -p $(pgrep -o -f "^python3 cache-warmer") | awk '{print "cache-warmer RSS now", int($1/1024), "MB"}'
               total        used        free      shared  buff/cache   available
Mem:           969Mi       924Mi        64Mi       468Ki       101Mi        45Mi
Swap:             0B          0B          0B
    PID %MEM   RSS CMD
   1867 62.3 619400 python3 cache-warmer.py 600
    451  2.1 21304 /usr/bin/google_osconfig_agent
cache-warmer RSS now 604 MB

free -h shows available down to tens of MB on a 1 GB machine, and the file cache (buff/cache) squeezed out to make room. ps --sort=-rss (RSS = memory really in use) points at the leaking program at about 600 MB, over 60 percent of all memory.

The OOM killer

Now a second program asks for 1.5 GB, more than the whole VM has. Linux has no swap to fall back on, so the kernel's out-of-memory (OOM) killer picks a process and ends it.

# Break it harder: ask for more memory than the VM has. The kernel's OOM killer steps in.
python3 -c "x = b'x' * (1500 * 1024 * 1024)"; echo "exit code $? (137 = killed by signal 9)"
sudo dmesg | grep -iE "out of memory|oom-kill" | tail -2 | sed 's/^\[[^]]*\] //'
bash:  1920 Killed                  python3 -c "x = b'x' * (1500 * 1024 * 1024)"
exit code 137 (137 = killed by signal 9)
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-8.scope,task=python3,pid=1920,uid=1000
Out of memory: Killed process 1920 (python3) total-vm:1549860kB, anon-rss:725244kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:1484kB oom_score_adj:0
OOM killer
A leak fills memory; a 1.5 GB request is killed by the OOM killer with exit code 137, and dmesg says why

The shell reports Killed and exit code 137 (128 + signal 9, SIGKILL). dmesg (the kernel's own log) tells you exactly what happened: Out of memory: Killed process ... (python3) with the memory it held. **Whenever a program dies with exit code 137 or "Killed" and no error of its own, check dmesg for the OOM killer first.** The same happens in containers and Kubernetes, where it shows as OOMKilled.

Fix it: stop the leak

# Fix it: stop the leaking program and check the memory comes back
pkill -f cache-warmer; sleep 3
free -h | head -2
               total        used        free      shared  buff/cache   available
Mem:           969Mi       331Mi       629Mi       468Ki       142Mi       638Mi

Memory is back. The real fix for a leak is in the code (or a restart schedule while it is fixed), plus a memory limit so one program cannot take the whole machine (for example MemoryMax= in a systemd unit or a container memory limit).

Break it: the disk is busy

A "backup job" writes 1 GB files to the disk again and again, with oflag=direct so the writes really go to the disk instead of the memory cache.

🎓⭐ Recommended by CareerByteCode · Live trainingCloud & DevOps Live Cohort Internshipโ‚น2,360See the training →
# Break it: a backup job that writes the disk flat out
nohup bash -c 'exec -a backup-job bash -c "while :; do dd if=/dev/zero of=$HOME/lab275/big.img bs=1M count=1000 oflag=direct status=none; done"' > /dev/null 2>&1 &
sleep 10
vmstat 1 4
iostat -dx 2 2 | awk '/^Device/ {h=$0} /^sda/ {l=$0} END {print h; print l}' | awk '{print $1, $2, $8, $9, $NF}' | column -t
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1      0 599952  14764 168516    0    0  1946  2804  273  305 16  4 72  8  0
 1  1      0 599952  14764 168516    0    0     0 104456  771  320  1  3 49 48  1
 1  1      0 599952  14764 168516    0    0     0 104448  740  285  0  2 50 48  0
 1  1      0 599952  14764 168516    0    0     0 104448  670  286  0  3 50 48  0
Device  r/s   w/s     wkB/s      %util
sda     4.00  415.50  104288.00  94.80

vmstat now shows b = 1 (one process blocked waiting for the disk), bo around 100,000 KB written per second, and wa (IO wait) near 50 percent: half the CPU time is spent waiting for the disk. iostat -dx confirms it from the disk's side: sda writes about 100 MB/s and is busy over 90 percent of the time (%util).

Find the IO hog

Why: pidstat -d shows disk reads and writes per process, and ps -C dd shows who started it (PPID is the parent process).

# Who is writing? pidstat -d shows disk use per process
pidstat -d 2 2 | grep -E "Average.*(UID| dd$)" | head -3
ps -o pid,ppid,cmd -C dd
Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
Average:     1000      2015      0.00 104187.53      0.00       0  dd
    PID    PPID CMD
   2015    1997 dd if=/dev/zero of=/home/you/lab275/big.img bs=1M count=1000 oflag=direct status=none

Fix it: ionice, or stop it

Why: ionice -c 3 puts a process in the idle IO class: it only gets the disk when nobody else needs it. Good for backups that must run during the day. Or stop it.

# Fix it: run it with idle IO priority (ionice), or stop it
for p in $(pgrep -f backup-job) $(pgrep -x dd); do sudo ionice -c 3 -p $p && echo "ionice idle for $p"; done
ionice -p $(pgrep -x dd | head -1)
pkill -f backup-job; pkill -x dd; sleep 2; rm -f big.img
vmstat 1 2 | tail -1
ionice idle for 1997
ionice idle for 2052
idle
 1  0      0 597088  15320 171656    0    0     0     0   50   48  0  0 100  0  0
Disk busy
IO wait near 50 percent and the disk over 90 percent busy; pidstat finds dd, ionice makes it idle priority, then it is stopped

Step 3: a 60-second health check

Why: when you log in to a slow server, run the same checks in the same order every time. This small script does the first minute for you: load, the last kernel messages, CPU/memory/IO, memory, disk space, and the top 3 CPU users.

# The 60-second check: one script you can run on any slow Linux server
cat > health.sh <<'SH'
#!/bin/bash
echo "== load";   uptime
echo "== kernel"; sudo dmesg | tail -2 | sed 's/^\[[^]]*\] //'
echo "== cpu+mem+io (vmstat)"; vmstat 1 3 | tail -2
echo "== memory"; free -m | head -2
echo "== disk";   df -h / | tail -1
echo "== top 3 cpu"; ps -eo pid,%cpu,comm --sort=-%cpu | sed -n 2,4p
SH
chmod +x health.sh && ./health.sh
== load
 20:49:13 up 5 min,  0 user,  load average: 0.69, 0.58, 0.27
== kernel
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-8.scope,task=python3,pid=1920,uid=1000
Out of memory: Killed process 1920 (python3) total-vm:1549860kB, anon-rss:725244kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:1484kB oom_score_adj:0
== cpu+mem+io (vmstat)
 1  0      0 598472  15340 171844    0    0     0     0   42   43  0  0 100  0  0
 1  0      0 598472  15340 171844    0    0     0     0   41   41  0  0 100  0  0
== memory
               total        used        free      shared  buff/cache   available
Mem:             969         336         584           0         182         633
== disk
/dev/sda1       9.7G  2.1G  7.1G  23% /
== top 3 cpu
      1  0.9 systemd
    207  0.5 systemd-journal
   2104  0.5 sshd
Health check
One script: load, kernel messages (the OOM kill is still there), vmstat, memory, disk and the top processes

Notice that the kernel section still shows the OOM kill from earlier. dmesg keeps history, which is exactly why it belongs in a health check.

POC: prove it works

  • Two runaway processes pushed CPU idle to 0 and the run queue to 3; pidstat and ps named them; renice and kill brought idle back to 100.
  • A leaking program took about 600 MB of 1 GB; a 1.5 GB request was killed by the OOM killer (exit 137) and dmesg explained it.
  • A write-heavy job pushed IO wait near 50 percent and disk use over 90 percent; pidstat -d found dd; ionice and kill fixed it.
  • A reusable 60-second health check script.

Quiz task: check what you learned

  1. A 4-CPU server has a load average of 3.5. Is it overloaded? (Answer: no, it is busy but under 4. At 8 or more, work is queueing.)
  2. vmstat shows wa at 60. What do you check next? *(Answer: the disk: iostat -dx 1 and pidstat -d 1 to find the process.)*
  3. A program dies with exit code 137 and no error message. First command? *(Answer: sudo dmesg | grep -i "out of memory".)*
  4. A nightly report must still run but slows the web app. What do you do? *(Answer: run it with nice -n 19 and ionice -c 3, or move it to a quiet hour.)*
  5. Which free column tells you how much memory is really left for new programs? *(Answer: available.)*

Clean up (delete everything)

Exit the VM (exit), then in Cloud Shell:

gcloud compute instances delete perf-lab --zone=us-east1-b --quiet 2>&1 | tail -1
echo "perf-lab VMs left: $(gcloud compute instances list --filter="name=perf-lab" --format="value(name)" | wc -l)"
echo "perf-lab disks left: $(gcloud compute disks list --filter="name~^perf-lab" --format="value(name)" | wc -l)"
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/perf-lab].
WARNING: The following filter keys were not present in any resource : name
perf-lab VMs left: 0
WARNING: The following filter keys were not present in any resource : name
perf-lab disks left: 0
Clean up
The perf-lab VM and its disk are deleted: 0 left

FAQ

**top or htop?** Both show the same data. htop is friendlier (colours, scrolling, tree view) but often not installed; top is everywhere. top -c shows full command lines.

Why is "free" memory so low on a healthy server? Linux uses spare memory as file cache to make things faster. That memory is given back at once when programs need it, which is why available matters, not free.

**What is steal time (st)?** Time your VM wanted the CPU but the cloud host was running another VM. Shared-core machines like e2-micro can show it under load. High steal all the time means you need a bigger or non-shared machine type.

Should I add swap to a cloud VM? A small swap file can stop sudden OOM kills on tiny VMs, but it makes the server slow instead of failing fast. For production, size memory properly and set memory limits.

How do I watch this over time instead of in the moment? sar (from sysstat) records history once it is enabled, and Cloud Monitoring with the Ops Agent graphs CPU, memory and disk for every VM (see What next).

Can these commands harm the server? The read-only ones (uptime, top, vmstat, iostat, pidstat, free, dmesg) are safe anywhere. Be careful with kill on production: make sure you know what the process is first.

Troubleshooting

You seeWhyFix
Load average above the number of CPUsCPU queue or processes stuck on IOvmstat 1: high us/sy = CPU, high b/wa = disk
A process dies with exit 137 or KilledThe OOM killer ended itsudo dmesg; fix the leak, add memory or a limit
iostat: command not foundsysstat is not installedsudo apt install sysstat
High wa, but pidstat -d shows nothingThe process ended, or writes are bufferedRun pidstat -d 1 while it happens; check iotop
ionice: option requires an argument -- 'p'pgrep found no process, so no PID was passedThe process already ended; check pgrep -a <name>
High st (steal)The shared host is busyUse a non-shared machine type for steady load

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Linux Disk and Storage: Mount, Resize and Fix a Full Disk⏱ 15 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterโ‚น499Build the projects →