Linux Disk and Storage: Mount, Resize and Fix a Full Disk
What you build: a data disk added, mounted and kept in fstab, filled to 100 percent, a deleted-but-open file found, the disk resized live from 10 to 20 GB, and an inode-full disk fixed.

✍️ Translate this blog into Hindi, Tamil or Telugu
"No space left on device" stops databases, breaks deployments and fills on-call nights. In this lab you learn Linux disks by living through that day on a free Google Cloud VM: you add a second disk, format and mount it, make it survive reboots, fill it to 100 percent, find what filled it, meet the deleted file that still eats space, grow the disk while the server keeps running, and finally hit the strange "disk full" that happens when the disk is almost empty.
Who it is for: beginners and junior engineers who can use a Linux terminal and want to handle disks without fear. Time: about 1 hour. Cost: almost free. The e2-micro VM is in Google Cloud's Always Free tier, and the extra standard disk (10 GB, then 20 GB) exists for about 15 minutes, which costs a tiny fraction of a cent. We delete everything at the end.
Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. Disk UUIDs, process ids and dates will differ on your machine.
🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime StarterβΉ499Build the projects →Part A: disks in plain English
Four layers sit between a disk and your files:

- Disk (block device): the raw storage, shown as
/dev/sda,/dev/sdb. In Google Cloud it is a persistent disk: network storage you can attach, detach and resize with one command. Real-life example: an empty plot of land. - Partition: a slice of a disk (
/dev/sda1). The boot disk has partitions. A data disk can use the whole disk without them, as we do. - Filesystem: the structure that turns raw space into files and folders (
ext4here).mkfscreates it. Real-life example: drawing the streets and house numbers on the plot. - Mount: attaching a filesystem to a folder (the mount point, here
/mnt/data). Linux has no drive letters: a disk appears as a folder in the one tree. - /etc/fstab: the list of mounts to repeat at every boot.
- Inode: one entry in the filesystem's index card box. Every file uses one. A filesystem has a fixed number of them, which is why "disk full" can happen with free space left.
The commands you will use: lsblk (list block devices), df -h (how full each filesystem is, in human sizes), du (how much space folders use), mkfs.ext4, mount, blkid (the disk's UUID), lsof (open files), resize2fs (grow an ext4 filesystem) and df -i (inodes).
Before you start
- A Google Cloud account (the free trial works) and a project.
- Open Cloud Shell (the
>_icon at the top right of the Google Cloud console) and set your project:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.comPart B: the lab
Step 1: a free e2-micro VM
Why: a small Linux server (Debian 12) with a 10 GB boot disk. --no-service-account --no-scopes gives it no Google Cloud permissions; this lab needs none.
gcloud compute instances create disk-lab --zone=us-east1-b --machine-type=e2-micro \
--image-family=debian-12 --image-project=debian-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard \
--no-service-account --no-scopes 2>&1 | tail -2
until gcloud compute ssh disk-lab --zone=us-east1-b --tunnel-through-iap --command="echo ssh ok" 2>/dev/null | grep -q ok; do sleep 10; done; echo "SSH works"NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
disk-lab us-east1-b e2-micro 10.142.0.18 EXTERNAL_IP RUNNING
SSH worksThis lab uses two terminal tabs on purpose. Steps marked (Cloud Shell) run in your first Cloud Shell tab, the one you used so far. From here on: open a second Cloud Shell tab (the + in the terminal tab bar) and in it open an interactive SSH session with the VM using the command below. Once you see the VM terminal prompt (you@disk-lab:~$), run every other step there, in ~/lab268. Keep both tabs open. (Or click Compute Engine > VM instances > SSH next to disk-lab in the console for the VM terminal.)
gcloud compute ssh disk-lab --zone=us-east1-b# On the VM (SSH tab)
mkdir -p ~/lab268 && cd ~/lab268Step 2: look at the disks
Why: always look before you change anything. lsblk shows the devices and partitions, df -hT how full each mounted filesystem is (with its type), and findmnt / what is mounted at /.
# What disks does this VM have, and how full are they?
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
df -hT -x tmpfs -x devtmpfs
findmnt /NAME SIZE TYPE FSTYPE MOUNTPOINTS
sda 10G disk
ββsda1 9.9G part ext4 /
ββsda14 3M part
ββsda15 124M part vfat /boot/efi
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 9.7G 2.1G 7.1G 23% /
efivarfs efivarfs 256K 26K 226K 11% /sys/firmware/efi/efivars
/dev/sda15 vfat 124M 12M 112M 10% /boot/efi
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda1 ext4 rw,relatime,discard,errors=remount-roOne disk, sda (10 GB). Its partition sda1 is the root filesystem / (ext4, 23 percent used). sda15 holds the boot files.
Step 3 (Cloud Shell): create and attach a data disk
Why: keep data on its own disk, apart from the operating system: you can grow it, snapshot it or move it to another VM. --device-name=data gives it a stable name inside the VM.
# (Cloud Shell) Create a 10 GB standard disk and attach it to the VM with the device name "data"
gcloud compute disks create disk-lab-data --size=10GB --type=pd-standard --zone=us-east1-b --format="value(name,sizeGb,status)" 2>&1 | grep -v WARNING
gcloud compute instances attach-disk disk-lab --disk=disk-lab-data --device-name=data --zone=us-east1-b 2>&1 | tail -1
gcloud compute instances describe disk-lab --zone=us-east1-b --format="table(disks[].deviceName,disks[].diskSizeGb)"Created [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/disks/disk-lab-data].
disk-lab-data 10 READY
Updated [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/disk-lab].
DEVICE_NAME DISK_SIZE_GB
['persistent-disk-0', 'data'] ['10', '10']Step 4: format and mount the new disk
Why: a new disk is empty bytes with no filesystem. /dev/disk/by-id/google-data is a stable name Google creates from the device name (sdb can change between boots). mkfs.ext4 -q -L data -m 0 makes an ext4 filesystem quietly, labels it data and keeps no space back for root (-m 0, fine for a data disk). Then we mount it on /mnt/data and give it to our user.
# On the VM (SSH tab)
# The new disk is empty: make a filesystem (ext4) on it and mount it on /mnt/data
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
ls -l /dev/disk/by-id/google-data
sudo mkfs.ext4 -q -L data -m 0 /dev/disk/by-id/google-data
sudo mkdir -p /mnt/data
sudo mount /dev/disk/by-id/google-data /mnt/data
sudo chown $USER: /mnt/data
df -hT /mnt/dataNAME SIZE TYPE FSTYPE MOUNTPOINTS
sda 10G disk
ββsda1 9.9G part ext4 /
ββsda14 3M part
ββsda15 124M part vfat /boot/efi
sdb 10G disk
lrwxrwxrwx 1 root root 9 Nov 18 20:18 /dev/disk/by-id/google-data -> ../../sdb
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdb ext4 9.8G 24K 9.8G 1% /mnt/data
Step 5: mount it at every boot (fstab)
Why: a mount command is forgotten at the next reboot. A line in /etc/fstab is not. Use the filesystem's UUID (a unique id that never changes) and the option nofail, so the VM still boots if this disk is ever missing. mount -a mounts everything in fstab: if it works now, it will work at boot.
# Mount it again after every reboot: a line in /etc/fstab, by UUID, with nofail
UUID=$(sudo blkid -s UUID -o value /dev/disk/by-id/google-data); echo "UUID=$UUID"
echo "UUID=$UUID /mnt/data ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
sudo umount /mnt/data && sudo mount -a && findmnt /mnt/dataUUID=d020934e-9c96-49c6-a79c-f980a2a14b63
UUID=d020934e-9c96-49c6-a79c-f980a2a14b63 /mnt/data ext4 defaults,nofail 0 2
TARGET SOURCE FSTYPE OPTIONS
/mnt/data /dev/sdb ext4 rw,relatimeA mistake in fstab without nofail can stop a VM from booting. Always test with umount then mount -a before you reboot.
Break it: fill the disk
We fill the data disk the way it happens in real life: an old 6 GB backup nobody deleted, a 3.5 GB application log, and then the app tries to write more.
# Break it: fill the disk until writes fail
cd /mnt/data && mkdir -p logs backups
fallocate -l 6G backups/old-backup.tar
fallocate -l 3500M logs/app.log
echo "one more line" >> logs/app.log && dd if=/dev/zero of=logs/more.log bs=1M count=600 status=none
df -h /mnt/datadd: error writing 'logs/more.log': No space left on device
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 9.8G 9.8G 0 100% /mnt/dataNo space left on device, 100 percent used, 0 available. Every program that writes to this disk now fails.
Find what filled it
Why: du -xh --max-depth=2 sums each folder (-x stays on this disk), sort -h sorts human sizes, and find -size +500M lists the big files themselves.
# Find what fills the disk: du, sorted by size
sudo du -xh --max-depth=2 /mnt/data 2>/dev/null | sort -h | tail -5
sudo find /mnt/data -xdev -type f -size +500M -exec ls -lh {} \; | awk '{print $5, $9}'16K /mnt/data/lost+found
3.8G /mnt/data/logs
6.1G /mnt/data/backups
9.8G /mnt/data
6.0G /mnt/data/backups/old-backup.tar
3.5G /mnt/data/logs/app.log
Break it again: a deleted file that still takes space
Someone deletes the 3.5 GB log to free space. But a program (here tail -f, standing in for an app that writes the log) still has the file open.
# Break it again: delete a big log that a program still has open. The space does NOT come back.
cd /mnt/data && rm -f logs/more.log
command -v lsof > /dev/null || sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq lsof > /dev/null 2>&1
tail -f logs/app.log > /dev/null &
sleep 1; rm logs/app.log; echo "app.log deleted"
df -h /mnt/data | tail -1
sudo lsof +L1 2>/dev/null | awk 'NR==1 || /deleted/ {print $1, $2, $4, $7, $10, $11}' | column -t
kill %1; sleep 1
df -h /mnt/data | tail -1app.log deleted
/dev/sdb 9.8G 9.5G 320M 97% /mnt/data
COMMAND PID FD SIZE/OFF NAME
tail 1431 3r 3670016014 /mnt/data/logs/app.log (deleted)
/dev/sdb 9.8G 6.1G 3.8G 62% /mnt/dataRight after rm, df still shows 97 percent used. A deleted file's space comes back only when the last program closes it. lsof +L1 lists open files that have no name any more: tail still holds app.log (deleted), 3.67 GB. When we stop that program, the space comes back (62 percent).
On a real server you restart the program, or empty the file instead of deleting it: truncate -s 0 app.log frees the space even while it is open. That is also why logrotate has the copytruncate option.
Fix it for good (Cloud Shell): resize the disk while it runs
Why: deleting files is a short-term fix. A Google Cloud persistent disk can be made bigger with one command, without stopping the VM. (Disks can only grow, never shrink.)
# (Cloud Shell) Fix it for good: make the disk bigger while the VM runs (no stop needed)
gcloud compute disks resize disk-lab-data --size=20GB --zone=us-east1-b --quiet > /dev/null 2>&1 && echo "disk resized"
gcloud compute disks describe disk-lab-data --zone=us-east1-b --format="value(sizeGb)"disk resized
20Grow the filesystem into the new space
Why: the disk is now 20 GB, but the filesystem still thinks it is 10 GB. resize2fs grows an ext4 filesystem, and it can do it while mounted ("on-line resizing").
# On the VM (SSH tab)
# Grow the filesystem into the new space (online, while mounted)
lsblk -o NAME,SIZE,MOUNTPOINTS /dev/disk/by-id/google-data
df -h /mnt/data | tail -1
sudo resize2fs /dev/disk/by-id/google-data 2>&1 | grep -v "^$"
df -h /mnt/data | tail -1🎓⭐ Recommended by CareerByteCode · Live trainingCloud & DevOps Live Cohort InternshipβΉ2,360See the training →NAME SIZE MOUNTPOINTS
sdb 20G /mnt/data
/dev/sdb 9.8G 6.1G 3.8G 62% /mnt/data
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/disk/by-id/google-data is mounted on /mnt/data; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 3
The filesystem on /dev/disk/by-id/google-data is now 5242880 (4k) blocks long.
/dev/sdb 20G 6.1G 14G 31% /mnt/data
lsblk sees 20 GB at once, df still says 9.8 GB, and after resize2fs it says 20 GB with 14 GB free. The app never stopped. (For the boot disk the steps are the same, plus growpart to grow the partition first; on Google Cloud images that happens by itself at the next boot.)
Break it: "No space left" on an almost empty disk
Now the strange one. We make a tiny 64 MB filesystem with only 2048 inodes and write lots of very small files, like a cache or a mail queue does.
# Break it: "No space left on device" while df shows free space = no inodes left
mkdir -p ~/lab268 && cd ~/lab268
fallocate -l 64M tiny.img && sudo mkfs.ext4 -q -F -N 2048 tiny.img
sudo mkdir -p /mnt/tiny && sudo mount -o loop tiny.img /mnt/tiny && sudo chown $USER: /mnt/tiny
for i in $(seq 1 2100); do echo x > /mnt/tiny/f$i 2>/dev/null || { echo "file $i: No space left on device"; break; }; done
df -h /mnt/tiny | tail -1
df -i /mnt/tiny | tail -1-bash: /mnt/tiny/f2038: No space left on device
file 2038: No space left on device
/dev/loop0 59M 2.1M 52M 4% /mnt/tiny
/dev/loop0 2048 2048 0 100% /mnt/tinyFile number 2038 fails with No space left on device, yet df -h says only 4 percent of the space is used. df -i tells the truth: 2048 of 2048 inodes used, 100 percent. Every file needs one inode, and there are none left.
Fix it: free the inodes
# Fix it: delete the many small files (or make the filesystem with more inodes)
cd ~/lab268
find /mnt/tiny -name 'f*' -delete
df -i /mnt/tiny | tail -1
sudo umount /mnt/tiny && rm tiny.imgfind: β/mnt/tiny/lost+foundβ: Permission denied
/dev/loop0 2048 11 2037 1% /mnt/tiny
Deleting the small files frees the inodes (11 left in use). (The Permission denied line is find politely skipping the root-owned lost+found folder.) For a filesystem that will hold millions of tiny files, make it with more inodes: mkfs.ext4 -N <number> or -i <bytes per inode>.
Step 6: unmount and take the disk out of fstab
Why: before removing a disk, unmount it and remove its fstab line. Otherwise the next boot waits for a disk that is gone (that is what nofail protects against).
# Before removing a disk: unmount it and take its line out of /etc/fstab
cd ~
sudo umount /mnt/data
sudo sed -i '\|/mnt/data|d' /etc/fstab
findmnt /mnt/data || echo "/mnt/data is not mounted"/mnt/data is not mountedPOC: prove it works
- A new 10 GB disk was formatted, mounted on
/mnt/dataand listed in fstab by UUID withnofail. - The disk filled to 100 percent;
duandfindfound the 6 GB backup and the 3.5 GB log. - A deleted log held 3.67 GB until its program closed it (
lsof +L1). - The disk grew from 10 to 20 GB and the filesystem followed with
resize2fs, with no downtime. - A filesystem ran out of inodes at 4 percent space used;
df -ishowed it and deleting files fixed it.
Quiz task: check what you learned
- Which command shows how full every mounted filesystem is? (Answer:
df -h;df -ifor inodes.) - You deleted a 5 GB log but
dfdid not change. What do you check? (Answer:sudo lsof +L1for a program still holding the deleted file; restart it.) - Why use a UUID and
nofailin/etc/fstab? (Answer: device names likesdbcan change;nofaillets the VM boot if the disk is missing.) - The disk is resized in Google Cloud but
dfshows the old size. What is missing? (Answer: grow the filesystem:sudo resize2fs <device>for ext4,xfs_growfs <mount point>for XFS.) No space left on devicewith 40 percent space free. What is the likely cause? (Answer: no inodes left; checkdf -i.)
Clean up (delete everything)
Why: the data disk is not free, so delete it first, in the Cloud Shell tab: detach it from the VM, then delete it. Then type exit in the VM tab to leave the VM, and delete the VM from the Cloud Shell tab.
# (Cloud Shell) Detach and delete the data disk
gcloud compute instances detach-disk disk-lab --disk=disk-lab-data --zone=us-east1-b 2>&1 | tail -1
gcloud compute disks delete disk-lab-data --zone=us-east1-b --quiet 2>&1 | tail -1Updated [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/disk-lab].
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/disks/disk-lab-data].gcloud compute instances delete disk-lab --zone=us-east1-b --quiet 2>&1 | tail -1
echo "disk-lab VMs left: $(gcloud compute instances list --filter="name=disk-lab" --format="value(name)" | wc -l)"
echo "disk-lab disks left: $(gcloud compute disks list --filter="name~^disk-lab" --format="value(name)" | wc -l)"Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/disk-lab].
disk-lab VMs left: 0
disk-lab disks left: 0
FAQ
Is resizing a disk safe while the VM runs? Yes, growing a persistent disk and then resize2fs (ext4) or xfs_growfs (XFS) are online operations. Take a snapshot first on anything important. Shrinking is not possible.
What is the difference between df and du? df asks the filesystem how much is used in total. du adds up the files it can see. When they disagree, look for deleted-but-open files (lsof +L1) or files hidden under a mount point.
Which disk type should I choose? pd-standard (cheapest, slower) for backups and logs, pd-balanced (the usual default) for most apps, pd-ssd for busy databases. Size also sets speed on persistent disks: bigger disks are faster.
What happens to an attached disk when I delete the VM? The boot disk is deleted with the VM by default. A disk you attached later is not, so it keeps costing until you delete it. Our lab deletes it on purpose in the clean up.
ext4 or XFS? Both are solid. Debian and Ubuntu use ext4 by default, Red Hat style systems use XFS. XFS can grow but never shrink, even offline.
How do I clean logs safely? journalctl --vacuum-size=200M for the system journal, logrotate for app logs, and truncate -s 0 file for a log a program still has open.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
No space left on device, df -h at 100 percent | The disk is full | sudo du -xh --max-depth=2 / to find it, then delete, move or resize |
No space left on device, df -h has room | Inodes are used up | df -i; delete small files; more inodes at mkfs |
df unchanged after deleting a big file | A program still has it open | sudo lsof +L1; restart that program |
mount: wrong fs type, bad option, bad superblock | No filesystem on the disk, or the wrong type | lsblk -f; mkfs.ext4 on a new, empty disk only |
| VM stuck at boot after an fstab change | A wrong fstab line without nofail | Use the serial console, fix fstab; always test with mount -a |
df shows the old size after a resize | The filesystem was not grown | sudo resize2fs /dev/disk/by-id/google-<name> |
What next
- New to the terminal? Linux for Cloud Beginners: 50 Commands on a Free VM.
- Get an alert before a disk is full: Monitor Linux Servers with Node Exporter and Ops Agent.
- Find and delete disks nobody uses: FinOps for Beginners: Read and Cut Your Cloud Bill.
- Store files outside the VM: Cloud Storage Deep Dive: Classes, Lifecycle and Signed URLs.
- More hands-on posts: ByteBlog. Practice in ByteLabs and see the courses.
Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.
⭐ Keep learning with CareerByteCode
Hand-picked to go deeper on this topicGet CareerByteCode Team's next blog by email
One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.
We email you once to confirm. By subscribing you agree to the privacy policy.
Related blogs
CareerByteCode TeamLinux Performance Troubleshooting: CPU, Memory and IO
What you build: a slow e2-micro diagnosed for real: a CPU hog, a memory leak and an OOM kill, a disk flat out, found with top, vmstat, pidstat and iostat and fixed with renice, ionice and kill.
#devops
CareerByteCode TeamSelf-Host a Git Server and CI Runner on a Free VM
What you build: Gitea and its CI runner (act_runner) on one free e2-micro VM; a push runs the tests, a bug turns CI red with the failing test shown, a revert turns it green, and gitea dump backs it al
#beginners
CareerByteCode TeamCron Jobs and Log Rotation on Linux
What you build: a disk report job run by cron every minute with its log rotated by logrotate, on a free VM, plus 3 real mistakes: relative paths, the % sign and an insecure log folder.
#beginners
CareerByteCode TeamLinux Users, Groups and sudo Explained
What you build: a shared Linux server for 3 people on a free VM: users, a developers group, a sudo rule that allows one job only, visudo catching a typo, the sudo log, and an account locked and remove
💬 Comments (0)
Top · NewestSign in to comment