Fix 'No space left on device' When df Shows Free Space

You're deploying to a server at 2am, the build kicks off, and every write fails with No space left on device. You run df -h and it cheerfully reports 40% used on /. That's the moment your brain short-circuits. I've watched senior engineers stare at that output for ten minutes before remembering the two things that cause it: inodes, or files that got deleted but never actually went away.

What's actually happening

On ext4, XFS, and most Unix filesystems, a disk has two pools of resources: data blocks and inodes. An inode stores the metadata for one file — permissions, timestamps, and pointers to the blocks holding its content. Every filesystem is created with a fixed inode count at mkfs time. If you burn through all of them, you can't create a new file even if you have terabytes of blocks sitting idle. df -h only shows blocks. It says nothing about inodes.

The second cause is sneakier. When a process opens a file and you delete it with rm, the directory entry vanishes but the inode and its blocks stay allocated until the last file descriptor closes. Nginx holding a logfile open is the classic case. Logrotate rotates the file, the old one gets rm'd, but nginx keeps writing to the phantom. Disk stays full. df reflects it; ls shows nothing. Feels like a haunting.

Fix it: step by step

Check inode usage first. This is a ten-second test and it's right more often than people expect.

df -i

Look at the IUse% column. If any filesystem is at 100%, you've found it. The usual suspects are /var/spool/postfix stuffed with queued mail, session files in /var/lib/php/sessions, or a directory someone's been dumping millions of 1KB files into (looking at you, /tmp).

Find the inode hog. Running this from the root of the offending filesystem, sorted so the worst offender sits at the top:

for d in /*; do echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"; done | sort -rn | head -20

That's slower than du but it counts entries, which is what you want. Drill down into the winner with the same loop.

Delete the junk. Be careful — don't rm -rf a live mail queue or session directory. For PHP sessions, find /var/lib/php/sessions -type f -mtime +1 -delete is safer than nuking the whole thing. For postfix, postsuper -d ALL deferred clears the deferred queue cleanly.

If inodes look fine, hunt deleted-but-open files. This is the other half of the puzzle.

lsof +L1 | head -30

+L1 lists files with a link count below 1 — i.e. deleted, but still held open. You'll see the process name, PID, and the size of the phantom file. Common offenders: nginx workers holding old access logs, a Java process keeping a rotated Tomcat log open, or a Postgres backend with a dropped temp file.

Release the handle. The clean fix is systemctl reload nginx or restarting whichever daemon is holding it. If it's a Java app and you can't restart it right now, you can truncate the file through /proc:

echo "" > /proc/<PID>/fd/<FD>

Find the FD number in the first column of lsof +L1 output. This zeroes the file and gives you your space back without killing the process. Ugly, but it works, and I've used it on production web servers more times than I'd like to admit.

On XFS, check for a separate problem. XFS allocates inodes dynamically, so inode exhaustion is rare. But No space left on device on XFS with space available usually means you hit the inode64/AG limit or the filesystem is genuinely fragmented. Run xfs_info /mountpoint and check the free block distribution.

If it still fails

Check for a read-only remount. Filesystems remount read-only after certain I/O errors. mount | grep ro, will show it. The kernel logs (dmesg -T | tail -50) will tell you why — usually a failing disk or a corrupted ext4 journal.
Container gotcha. If you're inside Docker or a Kubernetes pod, df may be reporting the host's filesystem, not the overlay. Check df -h / inside the container and compare to the host. A quota'd overlay2 layer can hit its limit while the host looks fine. docker system prune -a on the host is the usual remedy.
Reserved blocks. ext4 reserves 5% for root by default. A non-root process writing to a disk at 95% will see ENOSPC even though root sees free space. tune2fs -m 1 /dev/sda1 drops it to 1% if you know what you're doing.
Watch it live. If it's intermittent, inotifywait -m -r /path or iotop -oPa will show you what's churning. I've caught a runaway logging loop this way more than once.

Nine times out of ten, df -i or lsof +L1 solves it in under a minute. The other 10% is a disk that's about to die, and you'll want to see that in dmesg before it takes the box down with it.