You're trying to save a file or install a package and you get No space left on device. But df -h shows 20GB free. That's annoying. I've seen this on shared hosting, Docker containers, and busy web servers. The real problem isn't disk space — it's that your filesystem ran out of room for file records. Those records are called inodes.
Quick Fix: Find and delete the junk files
- Check inode usage on every mount. Run this:
df -iLook for a mount that shows IUse% at 100% (or close). That's your problem filesystem. Usually it's
/,/var, or/tmp. Note the Mounted on column — that's where you'll work.After running, you'll see a table. The line with
IUse%at 100 is the one. - Find the directory with the most files. Go to the root of that filesystem. For example, if
/is full:
Then run this to see which top-level folder has the most files:cd /for i in /[a-z]*; do echo "$i: $(find $i -type f 2>/dev/null | wc -l) files"; doneWait a few seconds. It'll print a list. The folder with the highest number is your junk pile.
You might see
/varor/tmpwith hundreds of thousands of files. - Drill down into that folder. Say it's
/var. Then:cd /var for i in */; do echo "$i: $(find $i -type f 2>/dev/null | wc -l) files"; doneRepeat until you find the specific subdirectory causing the mess. Common culprits:
/var/spool/postfix/maildrop— stuck emails/tmp— temp files from crashed processes/var/log— millions of small log files/home/user/.cache— browser or app cache
- Delete the files. Once you know the directory, remove them. For example, to delete all files in
/tmp:rm -rf /tmp/*If there are too many files,
rmmight choke. Usefindinstead:find /tmp -type f -delete 2>/dev/nullThis is faster and handles huge numbers.
After running, hit
df -iagain. You should see inode usage drop. Usually by 50-80%.
Why this works
Every file on a Linux filesystem needs an inode. The inode stores the file's metadata — permissions, timestamps, location on disk. The filesystem creates a fixed number of inodes when you format it. Once they're used up, you can't create new files, even if there's empty space.
This happens most often with:
- Mail servers that queue thousands of small emails
- Web servers with many tiny log files that never rotate
- Temp directories that fill up from scripts or bad cron jobs
- Docker containers with layers that create many small files
Deleting those small files frees up inodes instantly. The filesystem doesn't care about size — it only cares about count.
Less common variations
Inode leak from NFS or FUSE mounts
Sometimes an NFS mount or a FUSE filesystem (like sshfs) can appear to use inodes from the local filesystem. If you see inode usage high but can't find the files, check /proc/mounts for NFS mounts. Unmount them and check again. If the count drops, the remote server has the problem.
Filesystem with zero inodes created
Rare, but I've seen it on misconfigured ext4 volumes. Run tune2fs -l /dev/sda1 | grep Inode. If Inode count is 0, the filesystem was created with -i 0 or something broke. You'll need to backup and reformat with a proper inode count. This is a last resort.
Hidden files in directories with special characters
Some programs create files with spaces, newlines, or weird Unicode names. ls or rm might miss them. Use find /target -type f -print0 | xargs -0 rm to handle any filename safely.
Prevention: Stop it from happening again
- Set up log rotation. On most systems,
logrotatehandles this. Check/etc/logrotate.confand make sure your apps are in there. Addmaxage 7orrotate 4to keep log counts low. - Monitor inode usage. Add this to your monitoring tool (Nagios, Zabbix, Prometheus): check
df -iand alert when IUse% hits 80%. On a cron job you can rundf -i | awk '{if (NR>1 && $5+0 > 80) print $0}'. - Clean /tmp automatically. Most distros have a
tmpfiles.dconfig. Edit/usr/lib/tmpfiles.d/tmp.confand setAgeto something like1d. This deletes files older than one day. - Choose the right filesystem. If you're building a mail server or a cache-heavy app, use XFS or ext4 with a higher inode ratio. For ext4, format with
-i 4096(one inode per 4KB) instead of the default 16KB. This gives you 4x more inodes.
That's the whole thing. Next time you see No space left on device but df -h shows free space, check inodes first. You'll fix it in under two minutes.