Fix Linux 'No space left on device' When df Shows Free Space

1. Deleted files still held open by a process

This is the one I see all the time. You run df -h and there's 20GB free, but your app still screams No space left on device. The culprit is almost always a file that was deleted but a process still has it open. On Linux, the space isn't released until the file handle is closed. I had a client last month whose log rotation on a mail server did exactly this — the syslog daemon kept the old log open, and the disk filled up to 100% even though df showed the file was gone.

Find the deleted files

Use lsof to list open files marked as deleted:

sudo lsof +L1

Look for lines where the size column shows a big number. The +L1 flag only shows files with a link count of 0 (i.e., deleted but open).

Once you find the PID, check what it is:

ps aux | grep <PID>

Fix it

The clean fix is to restart the process holding the file. That releases the space. If it's a service like syslog or apache, just:

sudo systemctl restart rsyslog

Or kill the process (if it's safe):

sudo kill -9 <PID>

If you can't restart the process (maybe it's a database), you can truncate the file via /proc:

sudo truncate -s 0 /proc/<PID>/fd/<FD>

That instantly frees the space without killing anything. I've used that trick on a production PostgreSQL box where restarting wasn't an option.

2. Inode exhaustion

Sometimes you have plenty of disk space, but every directory is full of empty files. Each file eats an inode, and filesystems have a hard limit. When you hit it, you get No space left on device even though df -h shows free gigabytes. This happens a lot with mail spools or temp directories that accumulate millions of tiny files.

Check your inode usage

df -i

If the IUse% column is at 100%, that's your problem. The fix is finding and deleting the junk files.

Find the directory with too many files

Use find to count files per directory (this can take a while on huge trees):

sudo find / -xdev -type f | cut -d'/' -f2 | sort | uniq -c | sort -rn | head

Then drill into the worst offenders. For example, if /tmp is full of stale files:

sudo find /tmp -type f -atime +7 -delete

If you're on ext4, you can't add inodes after creation. The real fix is to either clean up or reformat with more inodes. But for most cases, deleting old files solves it.

3. A full tmpfs or /dev/shm

You're checking df -h on your root partition and it looks fine. But then the error pops up when writing to /tmp or /dev/shm. Those are often mounted as tmpfs, which uses RAM. If you fill up RAM, you get ENOSPC. I saw this on a CI server where a build script dumped gigabytes into /tmp and brought everything down.

Check tmpfs mounts

df -h /tmp /dev/shm

If they're at 100%, you need to either free space in them or increase their size in /etc/fstab. For a quick fix, delete old files:

sudo find /tmp -type f -atime +1 -delete

To increase the size, edit /etc/fstab and add a size parameter:

tmpfs /tmp tmpfs defaults,size=2G 0 0

Then remount:

sudo mount -o remount /tmp

But remember, tmpfs is volatile — it's cleared on reboot. If you need persistent space, move the data to disk.

Quick reference table

Start with the lsof check — that's the most common and the quickest to resolve. Then move to inodes and tmpfs. Nine times out of ten, one of these three is your culprit.