Server disk full from log files – quick fix & stop it coming back

Quick answer for advanced users

Run sudo journalctl --vacuum-time=7d on Linux or wevtutil el | % { wevtutil cl $_ } in PowerShell on Windows. Then configure log rotation so it never happens again.

Why this happens and when

Last month I got a panicked call from a client running a small e-commerce site. Their server was completely locked – couldn't even SSH in. Disk full. The culprit? Apache access logs. 45 GB of them. Standard web servers, databases, and system logs can grow out of control if nobody watches them. Windows Event Logs can do the same – I've seen 80 GB of IIS logs on a single drive. The default configuration on most OSes keeps logs forever. That's fine for a development box, but on a production server, it's a disaster waiting to happen.

Step-by-step fix

1. Check disk usage – On Linux run df -h or du -sh /var/log. On Windows use Get-PSDrive C in PowerShell. Find which drive is full.
2. Identify the biggest log files – Linux: sudo du -sh /var/log/* | sort -rh | head -10. Windows: Get-ChildItem C:\Windows\System32\winevt\Logs | Sort-Object Length -Descending | Select-Object Name, Length.
3. Clear the logs safely – For Linux: sudo journalctl --vacuum-time=7d clears systemd logs older than 7 days. For Apache/Nginx logs, truncate them: sudo truncate -s 0 /var/log/apache2/access.log. For Windows Event Logs: wevtutil el | % { wevtutil cl $_ } clears all event logs. Warning: This erases historical events – only do it if you don't need audit trails.
4. Free up space immediately – After clearing, run sudo systemctl restart rsyslog (Linux) or restart the Windows Event Log service. Then check disk again with df -h.

Alternative fixes if the main one fails

If logs are still growing after clearing, check for a stuck process writing to a log file. On Linux, sudo lsof +L1 /var/log shows deleted files still held open by processes. Restart the process (like Apache or syslog) to release the space. On Windows, a similar issue happens with IIS worker processes – restart IIS with iisreset. Another trick: compress logs instead of deleting them. On Linux, gzip /var/log/*.log shrinks them 90%. Just remember to rotate them later.

Prevention – set up log rotation

This is the real fix. Without rotation, you'll be back to square one in a month. On Linux, install logrotate if it's not there (sudo apt install logrotate on Ubuntu). Configure it in /etc/logrotate.conf. Example for Apache:

/var/log/apache2/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}

On Windows, use scheduled tasks to run a PowerShell script weekly. Here's a simple one:

$maxAge = 7
$logs = Get-ChildItem 'C:\inetpub\logs\LogFiles' -Recurse
$logs | Where-Object {$_.LastWriteTime -lt (Get-Date).AddDays(-$maxAge)} | Remove-Item -Force

Set this to run every Sunday at 2 AM via Task Scheduler. For Event Logs, use Group Policy to set maximum log size (default is 20 MB – bump it to 1 GB if you need retention, but never unlimited).

One more thing – check your backups

If logs are filling a drive, backup software might be failing silently. Had a client whose backup job kept writing huge temp files to the system drive because the backup target was full. Check your backup logs too. That's usually the most overlooked cause.

Pro tip: Set up a simple disk space alert. Nagios, Zabbix, or even a cron job that emails you when disk hits 90%. Costs nothing, saves a lot of headaches.