Quick answer for advanced users
Run df -i to check inode usage. If inodes at 100%, delete unnecessary files (especially in mail spools, temp directories, or Docker overlay2). If inodes are fine, check for a full /tmp in tmpfs, or a buggy filesystem needing fsck.
Why this happens
I had a client last month whose mail server kept throwing ENOSPC when trying to send emails. df -h showed 40% free on the root partition. Took me 10 minutes to realize the problem: millions of tiny files in the mail queue had eaten all inodes. Linux's standard df reports disk blocks (actual storage), but inodes are a separate pool — you can run out of inodes even with gigabytes free. This hits small-file-heavy workloads like mail servers, Docker containers with overlay2, or cron jobs that generate thousands of log files. The kernel returns ENOSPC when either blocks or inodes are exhausted. Sometimes it's a filesystem metadata corruption (less common, but happens after a crash) or a full tmpfs partition that tricks you.
Fix steps
- Check inode usage — Run
df -i | head -20. Look for 100% IUse% on the mount that's failing. That's your root cause more often than not. - Find the junk — If inodes full, use
find /your-mount-point -xdev -type f | wc -lto count files. Then drill into likely folders:/var/spool/mail,/tmp,/var/tmp,/var/log, and especially/var/lib/docker/overlay2if Docker is running. I once found 8 million leftover PHP session files in/tmpon a shared server. - Delete aggressively — Example:
find /tmp -type f -atime +7 -deleteremoves files older than 7 days. For Docker:docker system prune -a --volumes(but double-check you don't need those containers). On mail servers: clear the postfix queue or delete old spam. - Check for full tmpfs — Run
df -h | grep tmpfs. If/tmpis mounted as tmpfs and full, it'll report ENOSPC even if your root disk has space. Fix by moving tmp contents or resizing tmpfs withmount -o remount,size=2G /tmp. - Run filesystem check — If inodes look fine and tmpfs isn't the issue, the filesystem might be buggy. Unmount the partition (if possible) and run
fsck.ext4 -f /dev/sdX(replace with your device). For ext4, I've seen situations where the block bitmap gets corrupted and reports full. A fsck fixes it. Schedule downtime for this.
Alternative fixes if main steps fail
- Kernel bug workaround — On older kernels (pre-5.x), some filesystems like XFS had a bug where
dfreported free space but writes still failed. Upgrade the kernel if possible. Quick test:dd if=/dev/zero of=/path/testfile bs=1M count=1— if it fails with ENOSPC but df shows space, kernel bug is likely. - Mount remount — For NFS mounts, ENOSPC can happen if the server-side disk is full but the client's df doesn't reflect it. Run
mount -o remount /your/nfs/mountto refresh. - Use a rescue disk — If the root partition is full (blocks or inodes) and you can't even log in, boot from a live USB. Run fsck from there, then delete files from the mounted root.
Prevention tips
- Monitor inodes alongside space — Set up a cron job or use monitoring tools (Nagios, Zabbix) to alert when inode usage exceeds 80%. It's a silent killer.
- Set file limits — For mail servers, use
postfixlimits (e.g.,message_size_limit) to prevent queue bloat. For cron jobs, rotate logs aggressively —logrotatewith maxage 7 and maxsize 10M works. - Choose the right filesystem — If you know you'll have millions of small files (like a web cache or mail server), use XFS or btrfs over ext4. Ext4 limits inodes based on partition size at creation; XFS dynamically allocates inodes. I learned this the hard way after rebuilding a mail server twice.
- Schedule regular fsck — On non-production systems, run
tune2fs -c 30 /dev/sdXto force a check every 30 mounts. Catches corruption early.
“The real fix is understanding that 'no space' in Linux doesn't always mean you need a bigger disk — it means you need to look at what the kernel is actually running out of.”