Fix 'No space left on device' when disk isn't full

Quick answer

Run df -i / to see if your inode usage is at 100%. If yes, you're out of inodes, not disk space. Find and delete the files causing the bloat.

Why this happens

You're staring at df -h showing 40% used, but touch testfile returns No space left on device. Infuriating, right? I've been there. The culprit is almost always inode exhaustion. Every file and directory on an ext4 filesystem requires an inode—a small metadata structure that stores permissions, timestamps, and pointers to the actual data blocks. A filesystem has a fixed number of inodes created at format time. If you create millions of tiny files, you can burn through the inode table while the data blocks remain mostly empty. This often happens on mail servers (each email is a file), with Docker overlay2 storage, or when a misbehaving application writes endless zero-byte lock files.

Fix steps

Confirm it's inodes:
```
df -i /
```
Look at the IUse% column. If it's 100%, you found the problem.
Find the worst offenders. Search for directories with tons of files:
```
for d in / /var /tmp /home; do echo "$d: $(find $d -xdev -type f | wc -l) files"; done
```
Adjust the paths to your likely hotspots. On a typical server, check /var/spool, /tmp, and /var/log.
Drill into likely culprits. For mail queues:
```
find /var/spool/mail -type f | wc -l
```
For systemd journal logs, they're usually fine, but check /var/log/journal.
Delete unnecessary files. For email queues, purge old messages:
```
find /var/spool/mail -type f -mtime +30 -delete
```
Adjust the age to your retention policy. For Docker, clean up unused containers and images:
```
docker system prune -af
```
This alone freed 200k inodes on a client's server last month.
Check for hidden temp file storms. An app might be writing to /tmp and never cleaning up. Do:
```
find /tmp -type f | wc -l
```
If that's in the millions, clear old ones:
```
find /tmp -type f -atime +7 -delete
```

After deleting, verify with df -i / again. You should see the IUse% drop.

Alternative fixes if deletion isn't enough

Sometimes you can't delete files because they're critical. Options:

Reformat with more inodes. If this is a recurring issue on a data partition, back up, then recreate the filesystem with a higher inode ratio. For ext4:
```
mkfs.ext4 -i 16384 /dev/sdb1
```
That sets one inode per 16KB (default is usually 32KB), doubling the inode count. Only do this on a fresh partition—reformatting wipes data.
Move to XFS. XFS allocates inodes dynamically, so you can't run out the same way. If planning a migration, consider it.
Use tmpfs for /tmp. Add to /etc/fstab:
```
tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=1G 0 0
```
But note this clears tmp on reboot and uses RAM, so size it accordingly.

Prevention tip

Set up monitoring for inode usage alongside disk usage. On most systems, you can add a simple cron job that warns you early. Or use built-in tools like monit or Nagios. The real lesson: when you see ENOSPC, don't just check disk space. Make it a habit to run df -i first. I've seen admins waste an hour resizing partitions that weren't even full. Now you know better.