You're in the middle of a deploy, or maybe a cron job that's run for years, and suddenly it fails with No space left on device (ENOSPC). You check df -h and there's 20GB free. The first time this happens, it messes with your head. The culprit here is almost always inodes — the metadata structures that store file information on Linux filesystems.
Inodes hold permissions, timestamps, and pointers to the actual data blocks. Every file, directory, symlink, and even hard link consumes one inode. If you run out of inodes, the filesystem won't let you create anything new, even if there's plenty of free space. This typically happens on mail servers (millions of small spool files), tmp directories that never get cleaned, or apps that generate tons of tiny cache files.
How to check if it's inodes
Run df -i to see inode usage. If the IUse% column hits 100% (or close to it), that's your problem. Here's a sample snippet:
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 1234567 1234567 0 100% /var
If you see 100% on the mount that's failing, you're out of inodes. Don't bother resizing the filesystem unless you absolutely need to — the real fix is finding and deleting the junk files that ate them all.
Where to look first
Start with the usual suspects:
/tmp— old session files, PHP sessions, or leftover installs can pile up fast./var/tmp— same idea, but it survives reboots so it accumulates more./var/spool— mail queues, cron spools, or print spools are notorious for this./var/lib/docker— Docker containers create tons of small overlay files. If you're running Docker, check here first.
Use find to locate directories with massive numbers of files:
find /var -xdev -type f | wc -l
But that just gives you a total count. Better to drill down with ls | wc -l in specific dirs, or use du --inodes if your version supports it (it's in coreutils 8.22+). Example:
du --inodes /var/* | sort -n | tailThat lists the top offenders. If you don't have that, fall back to a loop:
for d in /var/*; do echo "$(find $d -xdev -type f | wc -l) $d"; done | sort -nThe fix: delete junk files
Once you've found the directory, it's just a matter of removing the junk. Be careful — don't delete anything that looks like real data. Start with the oldest files:
- Find old temp files:
find /tmp -type f -atime +30 -delete(adjust the time). - Clear stale sessions:
find /var/lib/php/sessions -type f -atime +30 -delete(path varies by distro). - If it's a mail spool, use your mail admin tools, don't just rm.
- Docker? Run
docker system prune -afto nuke unused containers and images.
After cleaning, run df -i again to confirm the inode count is down. Then retry your operation.
Preventing this from happening again
Set up log rotation and temp cleaning. On systemd systems, there's systemd-tmpfiles that can clear /tmp automatically — check /usr/lib/tmpfiles.d/tmp.conf. For mail spools, configure your MTA to delete old messages.
If this keeps biting you, consider adding a cron job that checks inode usage daily:
#!/bin/bash
THRESHOLD=90
CUR=$(df -i /var | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $CUR -gt $THRESHOLD ]; then
echo "Warning: /var inode usage at ${CUR}%" | mail -s "Inode alert" root
fi
What if it's not inodes?
If df -i shows plenty of inodes, but you still get ENOSPC, check for a different issue: a full mount that's not visible because something is hiding it. Mount points can have files underneath them — if you mount a new filesystem over a directory that already has files, those old files are invisible but still take up space.
Run mount to see all mounts, then check each one with df -h. Also look for deleted files that are still held open by processes:
lsof +L1That shows files with a link count of 0 — they're deleted but a process has them open. The space won't free until that process closes the file. Restart the process or kill it if it's stuck. Also, check for filesystem corruption with fsck if nothing else explains it. But honestly, 9 times out of 10, it's inodes.