CareerByteCode
ByteBlogby CareerByteCode

Linux Disk and Storage: Mount, Resize and Fix a Full Disk

What you build: a data disk added, mounted and kept in fstab, filled to 100 percent, a deleted-but-open file found, the disk resized live from 10 to 20 GB, and an inode-full disk fixed.

Linux Disk and Storage: Mount, Resize and Fix a Full Disk

✍️ Translate this blog into Hindi, Tamil or Telugu

"No space left on device" stops databases, breaks deployments and fills on-call nights. In this lab you learn Linux disks by living through that day on a free Google Cloud VM: you add a second disk, format and mount it, make it survive reboots, fill it to 100 percent, find what filled it, meet the deleted file that still eats space, grow the disk while the server keeps running, and finally hit the strange "disk full" that happens when the disk is almost empty.

Who it is for: beginners and junior engineers who can use a Linux terminal and want to handle disks without fear. Time: about 1 hour. Cost: almost free. The e2-micro VM is in Google Cloud's Always Free tier, and the extra standard disk (10 GB, then 20 GB) exists for about 15 minutes, which costs a tiny fraction of a cent. We delete everything at the end.

Every command below was run for real by the CareerByteCode Team on a fresh Google Cloud project, and every output and screenshot is the real result we got. Disk UUIDs, process ids and dates will differ on your machine.

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterβ‚Ή499Build the projects →

Part A: disks in plain English

Four layers sit between a disk and your files:

Disks in one picture
A disk is a device, a filesystem organises it, mounting attaches it to a folder, and fstab repeats the mount at every boot
  • Disk (block device): the raw storage, shown as /dev/sda, /dev/sdb. In Google Cloud it is a persistent disk: network storage you can attach, detach and resize with one command. Real-life example: an empty plot of land.
  • Partition: a slice of a disk (/dev/sda1). The boot disk has partitions. A data disk can use the whole disk without them, as we do.
  • Filesystem: the structure that turns raw space into files and folders (ext4 here). mkfs creates it. Real-life example: drawing the streets and house numbers on the plot.
  • Mount: attaching a filesystem to a folder (the mount point, here /mnt/data). Linux has no drive letters: a disk appears as a folder in the one tree.
  • /etc/fstab: the list of mounts to repeat at every boot.
  • Inode: one entry in the filesystem's index card box. Every file uses one. A filesystem has a fixed number of them, which is why "disk full" can happen with free space left.

The commands you will use: lsblk (list block devices), df -h (how full each filesystem is, in human sizes), du (how much space folders use), mkfs.ext4, mount, blkid (the disk's UUID), lsof (open files), resize2fs (grow an ext4 filesystem) and df -i (inodes).

Before you start

  1. A Google Cloud account (the free trial works) and a project.
  2. Open Cloud Shell (the >_ icon at the top right of the Google Cloud console) and set your project:
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID
gcloud services enable compute.googleapis.com

Part B: the lab

Step 1: a free e2-micro VM

Why: a small Linux server (Debian 12) with a 10 GB boot disk. --no-service-account --no-scopes gives it no Google Cloud permissions; this lab needs none.

gcloud compute instances create disk-lab --zone=us-east1-b --machine-type=e2-micro \
  --image-family=debian-12 --image-project=debian-cloud --boot-disk-size=10GB --boot-disk-type=pd-standard \
  --no-service-account --no-scopes 2>&1 | tail -2
until gcloud compute ssh disk-lab --zone=us-east1-b --tunnel-through-iap --command="echo ssh ok" 2>/dev/null | grep -q ok; do sleep 10; done; echo "SSH works"
NAME      ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
disk-lab  us-east1-b  e2-micro                   10.142.0.18  EXTERNAL_IP  RUNNING
SSH works

This lab uses two terminal tabs on purpose. Steps marked (Cloud Shell) run in your first Cloud Shell tab, the one you used so far. From here on: open a second Cloud Shell tab (the + in the terminal tab bar) and in it open an interactive SSH session with the VM using the command below. Once you see the VM terminal prompt (you@disk-lab:~$), run every other step there, in ~/lab268. Keep both tabs open. (Or click Compute Engine > VM instances > SSH next to disk-lab in the console for the VM terminal.)

gcloud compute ssh disk-lab --zone=us-east1-b
# On the VM (SSH tab)
mkdir -p ~/lab268 && cd ~/lab268

Step 2: look at the disks

Why: always look before you change anything. lsblk shows the devices and partitions, df -hT how full each mounted filesystem is (with its type), and findmnt / what is mounted at /.

# What disks does this VM have, and how full are they?
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
df -hT -x tmpfs -x devtmpfs
findmnt /
NAME     SIZE TYPE FSTYPE MOUNTPOINTS
sda       10G disk        
β”œβ”€sda1   9.9G part ext4   /
β”œβ”€sda14    3M part        
└─sda15  124M part vfat   /boot/efi
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/sda1      ext4      9.7G  2.1G  7.1G  23% /
efivarfs       efivarfs  256K   26K  226K  11% /sys/firmware/efi/efivars
/dev/sda15     vfat      124M   12M  112M  10% /boot/efi
TARGET SOURCE    FSTYPE OPTIONS
/      /dev/sda1 ext4   rw,relatime,discard,errors=remount-ro

One disk, sda (10 GB). Its partition sda1 is the root filesystem / (ext4, 23 percent used). sda15 holds the boot files.

Step 3 (Cloud Shell): create and attach a data disk

Why: keep data on its own disk, apart from the operating system: you can grow it, snapshot it or move it to another VM. --device-name=data gives it a stable name inside the VM.

# (Cloud Shell) Create a 10 GB standard disk and attach it to the VM with the device name "data"
gcloud compute disks create disk-lab-data --size=10GB --type=pd-standard --zone=us-east1-b --format="value(name,sizeGb,status)" 2>&1 | grep -v WARNING
gcloud compute instances attach-disk disk-lab --disk=disk-lab-data --device-name=data --zone=us-east1-b 2>&1 | tail -1
gcloud compute instances describe disk-lab --zone=us-east1-b --format="table(disks[].deviceName,disks[].diskSizeGb)"
Created [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/disks/disk-lab-data].
disk-lab-data	10	READY
Updated [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/disk-lab].
DEVICE_NAME                    DISK_SIZE_GB
['persistent-disk-0', 'data']  ['10', '10']

Step 4: format and mount the new disk

Why: a new disk is empty bytes with no filesystem. /dev/disk/by-id/google-data is a stable name Google creates from the device name (sdb can change between boots). mkfs.ext4 -q -L data -m 0 makes an ext4 filesystem quietly, labels it data and keeps no space back for root (-m 0, fine for a data disk). Then we mount it on /mnt/data and give it to our user.

# On the VM (SSH tab)
# The new disk is empty: make a filesystem (ext4) on it and mount it on /mnt/data
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
ls -l /dev/disk/by-id/google-data
sudo mkfs.ext4 -q -L data -m 0 /dev/disk/by-id/google-data
sudo mkdir -p /mnt/data
sudo mount /dev/disk/by-id/google-data /mnt/data
sudo chown $USER: /mnt/data
df -hT /mnt/data
NAME     SIZE TYPE FSTYPE MOUNTPOINTS
sda       10G disk        
β”œβ”€sda1   9.9G part ext4   /
β”œβ”€sda14    3M part        
└─sda15  124M part vfat   /boot/efi
sdb       10G disk        
lrwxrwxrwx 1 root root 9 Nov 18 20:18 /dev/disk/by-id/google-data -> ../../sdb
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb       ext4  9.8G   24K  9.8G   1% /mnt/data
A new disk
sdb appears with no filesystem, gets ext4, and is mounted on /mnt/data with 9.8 GB free

Step 5: mount it at every boot (fstab)

Why: a mount command is forgotten at the next reboot. A line in /etc/fstab is not. Use the filesystem's UUID (a unique id that never changes) and the option nofail, so the VM still boots if this disk is ever missing. mount -a mounts everything in fstab: if it works now, it will work at boot.

# Mount it again after every reboot: a line in /etc/fstab, by UUID, with nofail
UUID=$(sudo blkid -s UUID -o value /dev/disk/by-id/google-data); echo "UUID=$UUID"
echo "UUID=$UUID /mnt/data ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
sudo umount /mnt/data && sudo mount -a && findmnt /mnt/data
UUID=d020934e-9c96-49c6-a79c-f980a2a14b63
UUID=d020934e-9c96-49c6-a79c-f980a2a14b63 /mnt/data ext4 defaults,nofail 0 2
TARGET    SOURCE   FSTYPE OPTIONS
/mnt/data /dev/sdb ext4   rw,relatime

A mistake in fstab without nofail can stop a VM from booting. Always test with umount then mount -a before you reboot.

Break it: fill the disk

We fill the data disk the way it happens in real life: an old 6 GB backup nobody deleted, a 3.5 GB application log, and then the app tries to write more.

# Break it: fill the disk until writes fail
cd /mnt/data && mkdir -p logs backups
fallocate -l 6G backups/old-backup.tar
fallocate -l 3500M logs/app.log
echo "one more line" >> logs/app.log && dd if=/dev/zero of=logs/more.log bs=1M count=600 status=none
df -h /mnt/data
dd: error writing 'logs/more.log': No space left on device
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        9.8G  9.8G     0 100% /mnt/data

No space left on device, 100 percent used, 0 available. Every program that writes to this disk now fails.

Find what filled it

Why: du -xh --max-depth=2 sums each folder (-x stays on this disk), sort -h sorts human sizes, and find -size +500M lists the big files themselves.

# Find what fills the disk: du, sorted by size
sudo du -xh --max-depth=2 /mnt/data 2>/dev/null | sort -h | tail -5
sudo find /mnt/data -xdev -type f -size +500M -exec ls -lh {} \; | awk '{print $5, $9}'
16K	/mnt/data/lost+found
3.8G	/mnt/data/logs
6.1G	/mnt/data/backups
9.8G	/mnt/data
6.0G /mnt/data/backups/old-backup.tar
3.5G /mnt/data/logs/app.log
Disk full
100 percent full; du and find point at a 6 GB backup and a 3.5 GB log

Break it again: a deleted file that still takes space

Someone deletes the 3.5 GB log to free space. But a program (here tail -f, standing in for an app that writes the log) still has the file open.

# Break it again: delete a big log that a program still has open. The space does NOT come back.
cd /mnt/data && rm -f logs/more.log
command -v lsof > /dev/null || sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq lsof > /dev/null 2>&1
tail -f logs/app.log > /dev/null &
sleep 1; rm logs/app.log; echo "app.log deleted"
df -h /mnt/data | tail -1
sudo lsof +L1 2>/dev/null | awk 'NR==1 || /deleted/ {print $1, $2, $4, $7, $10, $11}' | column -t
kill %1; sleep 1
df -h /mnt/data | tail -1
app.log deleted
/dev/sdb        9.8G  9.5G  320M  97% /mnt/data
COMMAND  PID   FD  SIZE/OFF    NAME                    
tail     1431  3r  3670016014  /mnt/data/logs/app.log  (deleted)
/dev/sdb        9.8G  6.1G  3.8G  62% /mnt/data

Right after rm, df still shows 97 percent used. A deleted file's space comes back only when the last program closes it. lsof +L1 lists open files that have no name any more: tail still holds app.log (deleted), 3.67 GB. When we stop that program, the space comes back (62 percent).

On a real server you restart the program, or empty the file instead of deleting it: truncate -s 0 app.log frees the space even while it is open. That is also why logrotate has the copytruncate option.

Fix it for good (Cloud Shell): resize the disk while it runs

Why: deleting files is a short-term fix. A Google Cloud persistent disk can be made bigger with one command, without stopping the VM. (Disks can only grow, never shrink.)

# (Cloud Shell) Fix it for good: make the disk bigger while the VM runs (no stop needed)
gcloud compute disks resize disk-lab-data --size=20GB --zone=us-east1-b --quiet > /dev/null 2>&1 && echo "disk resized"
gcloud compute disks describe disk-lab-data --zone=us-east1-b --format="value(sizeGb)"
disk resized
20

Grow the filesystem into the new space

Why: the disk is now 20 GB, but the filesystem still thinks it is 10 GB. resize2fs grows an ext4 filesystem, and it can do it while mounted ("on-line resizing").

# On the VM (SSH tab)
# Grow the filesystem into the new space (online, while mounted)
lsblk -o NAME,SIZE,MOUNTPOINTS /dev/disk/by-id/google-data
df -h /mnt/data | tail -1
sudo resize2fs /dev/disk/by-id/google-data 2>&1 | grep -v "^$"
df -h /mnt/data | tail -1
🎓⭐ Recommended by CareerByteCode · Live trainingCloud & DevOps Live Cohort Internshipβ‚Ή2,360See the training →
NAME SIZE MOUNTPOINTS
sdb   20G /mnt/data
/dev/sdb        9.8G  6.1G  3.8G  62% /mnt/data
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/disk/by-id/google-data is mounted on /mnt/data; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 3
The filesystem on /dev/disk/by-id/google-data is now 5242880 (4k) blocks long.
/dev/sdb         20G  6.1G   14G  31% /mnt/data
Resized live
A deleted file still open holds 3.67 GB; then the disk grows to 20 GB and resize2fs grows the filesystem online

lsblk sees 20 GB at once, df still says 9.8 GB, and after resize2fs it says 20 GB with 14 GB free. The app never stopped. (For the boot disk the steps are the same, plus growpart to grow the partition first; on Google Cloud images that happens by itself at the next boot.)

Break it: "No space left" on an almost empty disk

Now the strange one. We make a tiny 64 MB filesystem with only 2048 inodes and write lots of very small files, like a cache or a mail queue does.

# Break it: "No space left on device" while df shows free space = no inodes left
mkdir -p ~/lab268 && cd ~/lab268
fallocate -l 64M tiny.img && sudo mkfs.ext4 -q -F -N 2048 tiny.img
sudo mkdir -p /mnt/tiny && sudo mount -o loop tiny.img /mnt/tiny && sudo chown $USER: /mnt/tiny
for i in $(seq 1 2100); do echo x > /mnt/tiny/f$i 2>/dev/null || { echo "file $i: No space left on device"; break; }; done
df -h /mnt/tiny | tail -1
df -i /mnt/tiny | tail -1
-bash: /mnt/tiny/f2038: No space left on device
file 2038: No space left on device
/dev/loop0       59M  2.1M   52M   4% /mnt/tiny
/dev/loop0       2048  2048     0  100% /mnt/tiny

File number 2038 fails with No space left on device, yet df -h says only 4 percent of the space is used. df -i tells the truth: 2048 of 2048 inodes used, 100 percent. Every file needs one inode, and there are none left.

Fix it: free the inodes

# Fix it: delete the many small files (or make the filesystem with more inodes)
cd ~/lab268
find /mnt/tiny -name 'f*' -delete
df -i /mnt/tiny | tail -1
sudo umount /mnt/tiny && rm tiny.img
find: β€˜/mnt/tiny/lost+found’: Permission denied
/dev/loop0       2048    11  2037    1% /mnt/tiny
Inodes
4 percent of the space used but 100 percent of the inodes; deleting the small files frees them

Deleting the small files frees the inodes (11 left in use). (The Permission denied line is find politely skipping the root-owned lost+found folder.) For a filesystem that will hold millions of tiny files, make it with more inodes: mkfs.ext4 -N <number> or -i <bytes per inode>.

Step 6: unmount and take the disk out of fstab

Why: before removing a disk, unmount it and remove its fstab line. Otherwise the next boot waits for a disk that is gone (that is what nofail protects against).

# Before removing a disk: unmount it and take its line out of /etc/fstab
cd ~
sudo umount /mnt/data
sudo sed -i '\|/mnt/data|d' /etc/fstab
findmnt /mnt/data || echo "/mnt/data is not mounted"
/mnt/data is not mounted

POC: prove it works

  • A new 10 GB disk was formatted, mounted on /mnt/data and listed in fstab by UUID with nofail.
  • The disk filled to 100 percent; du and find found the 6 GB backup and the 3.5 GB log.
  • A deleted log held 3.67 GB until its program closed it (lsof +L1).
  • The disk grew from 10 to 20 GB and the filesystem followed with resize2fs, with no downtime.
  • A filesystem ran out of inodes at 4 percent space used; df -i showed it and deleting files fixed it.

Quiz task: check what you learned

  1. Which command shows how full every mounted filesystem is? (Answer: df -h; df -i for inodes.)
  2. You deleted a 5 GB log but df did not change. What do you check? (Answer: sudo lsof +L1 for a program still holding the deleted file; restart it.)
  3. Why use a UUID and nofail in /etc/fstab? (Answer: device names like sdb can change; nofail lets the VM boot if the disk is missing.)
  4. The disk is resized in Google Cloud but df shows the old size. What is missing? (Answer: grow the filesystem: sudo resize2fs <device> for ext4, xfs_growfs <mount point> for XFS.)
  5. No space left on device with 40 percent space free. What is the likely cause? (Answer: no inodes left; check df -i.)

Clean up (delete everything)

Why: the data disk is not free, so delete it first, in the Cloud Shell tab: detach it from the VM, then delete it. Then type exit in the VM tab to leave the VM, and delete the VM from the Cloud Shell tab.

# (Cloud Shell) Detach and delete the data disk
gcloud compute instances detach-disk disk-lab --disk=disk-lab-data --zone=us-east1-b 2>&1 | tail -1
gcloud compute disks delete disk-lab-data --zone=us-east1-b --quiet 2>&1 | tail -1
Updated [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/disk-lab].
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/disks/disk-lab-data].
gcloud compute instances delete disk-lab --zone=us-east1-b --quiet 2>&1 | tail -1
echo "disk-lab VMs left: $(gcloud compute instances list --filter="name=disk-lab" --format="value(name)" | wc -l)"
echo "disk-lab disks left: $(gcloud compute disks list --filter="name~^disk-lab" --format="value(name)" | wc -l)"
Deleted [https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-east1-b/instances/disk-lab].
disk-lab VMs left: 0
disk-lab disks left: 0
Clean up
The data disk is detached and deleted, then the VM and its boot disk: 0 left

FAQ

Is resizing a disk safe while the VM runs? Yes, growing a persistent disk and then resize2fs (ext4) or xfs_growfs (XFS) are online operations. Take a snapshot first on anything important. Shrinking is not possible.

What is the difference between df and du? df asks the filesystem how much is used in total. du adds up the files it can see. When they disagree, look for deleted-but-open files (lsof +L1) or files hidden under a mount point.

Which disk type should I choose? pd-standard (cheapest, slower) for backups and logs, pd-balanced (the usual default) for most apps, pd-ssd for busy databases. Size also sets speed on persistent disks: bigger disks are faster.

What happens to an attached disk when I delete the VM? The boot disk is deleted with the VM by default. A disk you attached later is not, so it keeps costing until you delete it. Our lab deletes it on purpose in the clean up.

ext4 or XFS? Both are solid. Debian and Ubuntu use ext4 by default, Red Hat style systems use XFS. XFS can grow but never shrink, even offline.

How do I clean logs safely? journalctl --vacuum-size=200M for the system journal, logrotate for app logs, and truncate -s 0 file for a log a program still has open.

Troubleshooting

You seeWhyFix
No space left on device, df -h at 100 percentThe disk is fullsudo du -xh --max-depth=2 / to find it, then delete, move or resize
No space left on device, df -h has roomInodes are used updf -i; delete small files; more inodes at mkfs
df unchanged after deleting a big fileA program still has it opensudo lsof +L1; restart that program
mount: wrong fs type, bad option, bad superblockNo filesystem on the disk, or the wrong typelsblk -f; mkfs.ext4 on a new, empty disk only
VM stuck at boot after an fstab changeA wrong fstab line without nofailUse the serial console, fix fstab; always test with mount -a
df shows the old size after a resizeThe filesystem was not grownsudo resize2fs /dev/disk/by-id/google-<name>

What next

Prepared by the CareerByteCode Team from our tested lab, with AI help for the write-up.

Read next →Linux Performance Troubleshooting: CPU, Memory and IO⏱ 15 min

⭐ Keep learning with CareerByteCode

Hand-picked to go deeper on this topic
News, guides and updates from the CareerByteCode team
🏅 Expert writer

The CareerByteCode team writes about what is new on careerbytecode.in: courses, labs, exams, events and the community, which is free to join for influencers and content creators in tech.

Get CareerByteCode Team's next blog by email

One email a week when they publish. Add the weekly Best of ByteBlog too. No spam, unsubscribe with one click.

We email you once to confirm. By subscribing you agree to the privacy policy.

💬 Comments (0)

Top · Newest

Sign in to comment

Related blogs

🧪⭐ Recommended by CareerByteCode · ByteLabs project bundleCloud and DevOps Realtime Starterβ‚Ή499Build the projects →