ENOSPC: No Space Left on Device but df Shows Free

Quick answer: Run df -i, df -h /tmp, and lsof +L1 — the culprit is almost always inode exhaustion, a full separate /tmp, or deleted files a process is still holding open.

You're trying to write a log file, save a database dump, or install a package, and the shell spits back No space left on device. You check df -h and see 40% used on the volume. Plenty of room. So you try again. Same error. That gap between what df reports and what the kernel actually allows is what trips people up. On Linux and the BSDs, "free disk space" is a much narrower idea than most people assume. A filesystem can run out of inodes long before it runs out of bytes. A process can keep a deleted 8 GB file pinned in memory because it never closed the file descriptor. Or you've got a separate /tmp mount that filled up while / looks fine.

I've seen this bite people on CentOS 7 boxes running a mail queue, on Ubuntu 20.04 containers running a Node app that writes millions of tiny cache files, and on FreeBSD jails where the operator forgets /var is its own partition. The error message doesn't care. It just says no space, and it's up to you to figure out which kind of space it means.

Step-by-step: find the real cause

Check inode usage first. This is the most common trap and the fastest to rule out. Run:
```
df -i
```
You'll see a table with IUsed, IFree, and IUse% columns. If IUse% is at 100% on any mounted filesystem, that's your problem. Each file, directory, symlink, and socket takes one inode. A directory holding two million 2 KB files fills up inodes while using maybe 4 GB of actual space. Mail queues, session directories, and container overlay layers are the usual offenders.
Check every mount, not just the one you think you're on. If you're writing to /tmp, it might be a tmpfs or its own partition. Run:
```
df -h
```
Look for the mount point that contains your target path — not the one that looks most used. Use df -h /path/to/file to get the exact filesystem. A tmpfs /tmp capped at 2 GB fills fast when something like ImageMagick writes temp files there.
Hunt for deleted files still held open. A process that opens a file and then unlinks it keeps the blocks allocated until the file descriptor closes. Common with log rotation that renames instead of truncates, or a database that rotated its WAL. Run:
```
lsof +L1
```
or on systems without lsof:
```
ls -l /proc/*/fd 2>/dev/null | grep deleted
```
If you find a large deleted file tied to a PID, restart that service. The space comes back immediately.
Look at reserved blocks. ext2/3/4 filesystems reserve 5% of the volume for root by default. If you're a non-root user and the volume is 95% full, you'll get ENOSPC even though df shows a few GB free. Check:
```
tune2fs -l /dev/sda1 | grep -i reserved
```
If the math doesn't work out for your workload, reduce it (this example drops reserved to 1%):
```
tune2fs -m 1 /dev/sda1
```
Don't set it to zero on the root filesystem. You'll regret it the first time a runaway log fills the disk and root can't even log in.
Check the process's view of the world. In containers and chroots, your shell and your service may live on different mounts. Confirm with:
```
cat /proc/<PID>/mountinfo
```
Compare with your own /proc/self/mountinfo. If the service is writing to an overlay or bind mount you can't see from the host shell, this explains the mismatch.

Alternative fixes if the main ones don't work

Filesystem errors. Sometimes the filesystem is corrupt and df reports stale numbers. Run dmesg | tail -50 and look for EXT4-fs error or I/O error. If you see them, unmount and run fsck before trusting anything else on that volume.
Quota limits. On shared hosts, you may be hitting a per-user quota, not a filesystem full condition. Check with quota -s (Linux) or quota -v (BSD). The error message is identical.
Read-only remount. If the kernel remounted the filesystem read-only after an error, every write returns ENOSPC even on an empty disk. Check with mount | grep ro,. Reboot or remount rw after fixing the underlying issue.
NFS weirdness. Stale NFS handles and full server-side exports masquerade as local ENOSPC. Run strace -e trace=write,openat df -h /mnt/nfs and watch which syscall fails.

Prevention

Set up monitoring that tracks both byte usage and inode usage, plus a separate alert on lsof +L1 output above a threshold. Nagios, Zabbix, and Prometheus node_exporter all cover this — pick one and actually configure the inode check, because the default templates often skip it. For log rotation, use copytruncate or send a signal to the daemon to reopen its log, so you're never left with a deleted-but-open file eating your disk. And on any new system, run df -h and df -i side by side once a quarter. Five seconds now saves you a 2 a.m. page later.