fix 'No space left on device' despite free space

You're running a build job on a ext4 filesystem, or maybe a cron script that writes thousands of small files to /tmp. The error pops up: No space left on device. You run df -h and see 20 GB free. You scratch your head. The write fails anyway. This isn't a disk space problem—it's an inode problem.

What's actually happening here is the filesystem ran out of inodes. Inodes are the metadata structures that store file names, permissions, timestamps, and pointers to data blocks. Every file, directory, symlink, and socket consumes one inode. When you create a filesystem with mkfs.ext4, it pre-allocates a fixed number of inodes based on the partition size and the bytes-per-inode setting. Once those are used up, the kernel rejects new file creations with ENOSPC even though data blocks remain empty.

This is a sneaky failure because it's invisible if you only check bytes. The fix depends on whether you're at the inode limit or just have a burst of temporary files. Here's how to diagnose and resolve it.

Step 1: Confirm it's inodes

Run df -i to see inode usage per filesystem:

df -i

Look at the IUse% column. If it's at 100% on the mount where the error occurs, that's your culprit. For example, you'll see something like:

/dev/sda1      1048576  1048576       0 100% /var

Zero free inodes right there. Also check df -h on the same mount—if bytes are low too, you might have a double whammy, but the inode exhaustion is what's blocking your write.

Step 2: Find the inode hog

You need to locate directories with a huge number of files. The find command can count them, but be careful—it scans the whole tree and can take a while on large filesystems.

find /var -xdev -type f | wc -l

That gives a total count. To find the worst subdirectories, use du --inodes if you have GNU coreutils 8.22 or later (most distros do):

du --inodes /var/* | sort -n | tail -20

This lists subdirectories sorted by inode usage. Common culprits: /var/spool (mail queues), /tmp (temp files), /var/tmp, or a logging directory with logrotate that never runs. If you see a temp dir with millions of files, you found your problem.

Step 3: Delete the offending files

If it's a temp directory, clearing it is straightforward:

find /tmp -xdev -type f -delete

Or for a specific directory like /var/spool/postfix/maildrop:

find /var/spool/postfix/maildrop -type f -delete

But sometimes you can't just delete—maybe the files are needed but the directory structure is bloated. In that case, you have to move some files to another filesystem or archive them with tar and remove the originals. The key is to free up inodes, not necessarily bytes.

Step 4: If it keeps happening, address the root cause

Deleting once is a band-aid. If your application generates endless small files (think PHP session files, Docker containers, or a misconfigured logrotate), you need to fix the generation rate. For logrotate, check your config in /etc/logrotate.d/ and ensure rotate and maxage are set. For temp files, consider using systemd-tmpfiles to clean /tmp regularly—it already runs by default on most distros, but you can tune it in /etc/tmpfiles.d/.

If you're genuinely out of inodes because the filesystem was created with too few, you have two options. The practical one is to back up and recreate the filesystem with a lower bytes-per-inode value. For a partition that's 1 TB and will hold lots of small files, you'd do:

mkfs.ext4 -i 16384 /dev/sdb1

That gives you roughly 64 million inodes instead of the default 16 million. But you can't change inode count on an existing ext4 filesystem—it's fixed at creation time. So this means a reformat and a restore. Not fun, but sometimes necessary.

Step 5: Check tmpfs and other memory-backed mounts

Another scenario: the error appears in /dev/shm or /run, which are tmpfs mounts backed by RAM. These have their own inode limits, but they're rarely the issue because they're huge. More likely, you filled up the tmpfs with data. Check df -h for those mounts—if /dev/shm is at 100%, the kernel gives you the same ENOSPC. The fix is to either free memory or increase the tmpfs size in /etc/fstab by adding size=2G to the mount options.

What if it still fails?

If you freed inodes but the error persists, don't assume you're done. Sometimes the write fails for a different reason that masks as ENOSPC. Here's what to check next:

• Check reserved blocks. ext4 reserves 5% of space for root. If you're a non-root user and the filesystem is at 95% full, you get ENOSPC even though there's technically free space. As root, you can use tune2fs -m 1 /dev/sda1 to lower the reserve to 1%.
• Look at the mount's file creation flags. If a directory has the immutable or append-only attribute (lsattr), writes fail with EPERM, not ENOSPC, but it's worth ruling out.
• Verify the process has write permission. A read-only filesystem or a permission issue can produce a generic error. Check mount | grep ro and id to see your user.
• Check for quota limits. If you're on a multi-user system with quotas, quota -vs will show if you've hit your inode or block quota. That fails with EDQUOT, but some apps report it as ENOSPC.

The inode exhaustion is the most common cause of ENOSPC with free bytes, and it's easy to diagnose with df -i. Once you know that, the fix is just a matter of deleting enough files or creating a filesystem that matches your file count needs. Don't waste time resizing partitions—that won't help. You need fewer files or more inodes.