Fix 'No space left on device' When df Says You Have Space

You've got a server throwing No space left on device while df -h swears up and down that you've got plenty. Annoying, sure — but it's one of the most diagnosable errors in Linux once you know where to look.

Two things cause this 95% of the time: you've run out of inodes, or you've got deleted files still being held open by running processes. Let's find out which.

Step 1: Check inodes first

Run this:

df -i

If IUse% is at 100% on any mount, that's your answer. You've got free blocks but zero free inodes, so the filesystem can't create new files. Doesn't matter if you have terabytes of space left — no inode, no file.

Real-world trigger: a mail server with 4 million tiny files in /var/spool/postfix, or a PHP app writing session files by the million into /tmp without cleanup. Also common on container hosts where /var/lib/docker accumulates layers on an inode-constrained ext4 volume.

Step 2: Find what's eating inodes

On the affected filesystem, drill down by directory count:

for i in /var/*; do echo "$i: $(find $i -xdev -print 2>/dev/null | wc -l)"; done

Then descend into the worst offender. If it's /var/spool or a mail queue, you can clear old messages. If it's a session directory, add cleanup. If it's Docker, docker system prune helps, but the real fix is moving /var/lib/docker to its own large filesystem.

You can't add inodes to an existing ext4 filesystem. You have to delete files or rebuild with mkfs.ext4 -N <count>. Plan for the rebuild — it's worth doing.

Step 3: If inodes are fine, look for deleted-but-open files

This one trips people up. A process opens a file, you rm it, but the process still holds the file descriptor. The kernel keeps the data on disk until the last descriptor closes. df doesn't show it, but du doesn't either. That kind of ghost space.

Find them:

lsof +L1 2>/dev/null | awk '{print $1, $2, $7, $9}' | sort -k3 -n -r | head -20

You want the SIZE/OFF column (bytes). Anything in the gigabytes is worth attention. Common offenders: a long-running rsyslogd whose log got rotated and deleted, a Java app holding onto a logfile, a crashed PostgreSQL backend still holding a huge temp file.

Step 4: Recover the space

For most processes, a SIGHUP or a graceful restart will close the descriptor and free the space:

kill -HUP <pid>
# or if that doesn't work:
systemctl restart <service>

Don't do kill -9 unless you have to. Graceful shutdown lets the app flush buffers and close descriptors cleanly. If it's a database, plan the restart — a hung backend holding 200GB of WAL isn't a good time for an unplanned bounce.

If the process can't be restarted (a customer-facing Java service, for example), truncate the file through /proc:

# Find the fd number from lsof output
: > /proc/<pid>/fd/<fdnum>

The : > (null redirect) truncates the file to zero without closing it. The process keeps running, the space comes back. This is the surgical version, and it's beautiful when you're in a war room at 3 AM.

Why this happens at all

Linux separates unlinking a file from deleting its data. When you rm, you remove the directory entry, but the inode and blocks stick around until every open file descriptor is closed. That's a feature, not a bug — it's why programs can keep reading a file after someone deletes it mid-read. But it means disk usage and df output can diverge in ways that look impossible.

The inode thing is simpler. ext4, XFS, and most traditional filesystems allocate a fixed number of inodes at mkfs time based on bytes-per-inode. Ext4 defaults to 16KB per inode, which is plenty for normal use. But if you're storing millions of 1KB files, you burn through inodes way before you burn through blocks.

Less common variations

Reserved blocks on ext4. By default, ext4 reserves 5% of blocks for root. A non-root process can hit ENOSPC while df shows plenty — because that 5% isn't available to it. Check with tune2fs -l /dev/sdX | grep Reserved. On huge data volumes, drop it to 1% with tune2fs -m 1 /dev/sdX. Don't drop it to 0 on the root filesystem — you want that buffer for recovery.

tmpfs limited by RAM. /tmp and /run are often tmpfs, sized to half of RAM by default. Writing a big file to /tmp when RAM is already tight throws ENOSPC even though df on the root FS is fine. Check df -h /tmp specifically, not just /.

Quota on the user or group. quotacheck-based quotas will throw ENOSPC when the user hits their limit, regardless of actual free space. Run repquota -a to see who's over.

XFS vs ext4 behavior. XFS allocates inodes dynamically, so inode exhaustion is rare. But XFS can throw ENOSPC on a nearly-full filesystem if it can't find a contiguous extent for a new file — even with a few megabytes free. Solutions: leave more headroom (XFS likes 5-10%), or defrag with xfs_fsr.

Read-only remount after errors. If the filesystem hit an I/O error, the kernel often remounts it read-only (RO). Any write returns ENOSPC because the FS is read-only. dmesg | tail will show EXT4-fs error or remounting filesystem read-only. That's a hardware/storage problem, not a space problem. Fix the disk first.

Sparse files and thin volumes. LVM thin pools and btrfs can overcommit. The volume reports free space, but the underlying pool is full. Check lvs -a for thin pool usage, and btrfs fi df /mount for btrfs.

Prevention

Monitoring that only checks df -h will miss both of the top causes of this error. Do these things:

• Alert on df -i too, not just df -h. Set thresholds at 85% for both.
• Alert on lsof +L1 aggregated size. Any process holding >1GB of deleted files gets an alert. Nagios, Zabbix, Prometheus blackbox — pick your poison, but wire it up.
• Set log rotation with copytruncate for apps that don't reopen logs on HUP. This is the /etc/logrotate.d/ option that catches 80% of deleted-but-open log files before they become a problem.
• For high-file-count workloads (mail, containers, CI artifacts), build filesystems with a lower bytes-per-inode value. mkfs.ext4 -i 4096 gives 4x more inodes.
• Never drop reserved blocks to 0 on /. That 5% is your emergency landing strip when things go wrong.
• For XFS, keep 10% free. It's not paranoia, it's how XFS prefers to allocate.

The next time No space left on device pops up with free space showing, you'll spend 30 seconds on df -i and lsof +L1 instead of an hour guessing. Bookmark those two commands. They save careers.