ENOSPC: No Space Left on Device But df Shows Free

Your disk looks half empty but Linux still throws ENOSPC. The usual cause is deleted files held open by a running process, or inodes running out instead of blocks.

It's 2am. Your app just started throwing write failed: No space left on device on a box where df -h cheerfully reports 40% used. You restart the service, it works for ten minutes, then dies again. This happens on ext4, XFS, and btrfs alike, and it almost always comes down to one of two things: files that were deleted but are still held open by a running process, or inodes running out while block space sits there looking pretty.

Both have the same symptom. Both have completely different fixes. Let's figure out which one you're staring at.

Why df and reality disagree

On a normal filesystem, when you rm a file, the kernel drops the directory entry but doesn't free the blocks until every file descriptor pointing at that file is closed. If nginx, a Java process, or a PostgreSQL backend still has the file open, the space stays allocated. df counts it. du doesn't, because du walks directory entries and the entry is gone. That mismatch is your fingerprint.

The classic real-world trigger: your log rotation script runs rm /var/log/app/access.log instead of sending the process a SIGHUP or using logrotate with copytruncate. The app keeps writing to the same inode it opened days ago. Blocks pile up against a file that no longer has a name.

The second cause is inode exhaustion. A directory with ten million 200-byte session files will eat every inode on the filesystem long before it fills the disk. df -h shows free space. df -i shows 100% inode use. Writes fail with ENOSPC anyway, because to create a file you need an inode, not just blocks.

Fix it step by step

Confirm which failure mode you're in. Run both commands and compare:
```
df -h /
df -i /
```
If df -h shows free blocks and df -i shows IUse% at 100, jump to step 5. Otherwise keep going.
Find deleted files still held open. The lsof +L1 flag lists files with a link count below 1, which is the kernel's way of saying "this has no directory entry but someone still has it open."
```
sudo lsof +L1
```
You'll get output like this:
```
COMMAND   PID   USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
nginx   14823   root    4w   REG  253,0  463190016     0  131086 /var/log/nginx/access.log (deleted)
```
That 463 MB is real space, sitting on disk with no name attached. If you see several of these, add up the SIZE/OFF column. The total is often exactly the "missing" space.
Free the space without killing anything. The clean way is to truncate the file through its file descriptor in /proc. Take the PID and FD from the previous step:
```
sudo truncate -s 0 /proc/14823/fd/4
```
After that, re-run df -h. You should see the free space jump by roughly the size you saw in lsof. If the number barely moves, you truncated the wrong FD. Check ls -l /proc/14823/fd/4 first to make sure it points where you think.
Restart the process if truncate isn't safe. For databases and anything holding a write lock, don't truncate. A systemctl reload nginx or systemctl restart postgresql closes every file descriptor and the kernel reclaims the blocks the moment the last FD drops. Expect a short outage. Plan for it.
Hit the inode wall? Find the swarm. If df -i was the culprit, hunt the directory that's spawning millions of tiny files:
```
for d in /var/* /home/* /tmp/*; do echo "$(find $d -xdev -type f 2>/dev/null | wc -l) $d"; done | sort -rn | head
```
The top entry is usually the offender. It's often a session directory, a mail spool with millions of queued messages, or a cache directory nobody pruned. Delete with find /path -type f -mtime +30 -delete and re-check df -i. You should see the count dropping immediately.
Check ext4 reserved blocks if you're truly out. On ext4, 5% of the filesystem is reserved for root by default. A normal user can hit ENOSPC while root still has a few GB. Look at the Use% column: if it's at 100 but root can still write, this is your culprit. Tune it down on data volumes:
```
sudo tune2fs -m 1 /dev/sda1
```
Don't do this on the root filesystem of a production box. Those reserved blocks exist so root can still log in and fix things when the disk fills. Use them, don't steal them.

If it still fails after all that

Check for a full /tmp mounted as a separate tmpfs. Docker and systemd both use it heavily. df -h /tmp might show 100% while the root filesystem is fine, and your failing process is writing temp files there.
Look at mounted overlay filesystems. If you're on Docker or Kubernetes, the container's writable layer has its own quota. df -h inside the container lies. Check the host.
Run dmesg | tail -50 and look for EXT4-fs error or XFS messages. A corrupted filesystem can report free space that isn't actually allocatable.
Verify your mount options. mount | grep " / " should show rw, not ro. A read-only remount after an I/O error produces the same ENOSPC-looking failures on some apps.
Check quotas on shared servers: quota -s as the affected user. A user over their block quota sees ENOSPC on a filesystem with petabytes free.

The real fix for the deleted-file case isn't a better cleanup script. It's fixing the thing that deletes logs out from under a running process. Switch logrotate to copytruncate or wire your app to reopen log files on SIGHUP. Otherwise you'll be right back here in three weeks at 2am.