Journal log rotation failure fix (no data loss)

Quick answer: Check disk space, then edit /etc/systemd/journald.conf, set SystemMaxUse=500M, MaxFileSec=1month, then sudo systemctl restart systemd-journald.

Here's why this happens. journald logs everything — boots, cron, salt, ssh attempts. If you never rotate them, the log folder grows until it chokes the system. I had a client last month whose Ubuntu 22.04 server stopped accepting SSH connections because journald filled the root partition. The box was still running, but nothing could write new logs or even a shell prompt. The fix? Not a reset. Just a configuration tweak.

Step 1: Check current disk usage

df -h /var/log/journal

If that folder is using more than 50% of your root partition, you've found the problem. On a typical cloud VM with 20GB root, journal should take under 1GB.

Step 2: Check current journal stats

journalctl --disk-usage

That shows total used. For example: Archived and active journals take up 1.2G in the file system. That's already too high if your root is small.

Step 3: Edit the config

Open /etc/systemd/journald.conf with sudo. Look for these lines (they're commented out by default):

#SystemMaxUse=
#MaxFileSec=

Change them to:

SystemMaxUse=500M
MaxFileSec=1month

That tells journald: keep max 500MB of logs total, and each file lives at most one month. Older files get deleted when new ones need space. This works on all distros — RHEL 8, Debian 11, Ubuntu 22.04, Fedora 37, same deal.

Step 4: Restart journald

sudo systemctl restart systemd-journald

This doesn't delete existing logs. It just tells daemon to apply the new limits. It will shrink the log folder gradually as old files expire and new ones get rotated under the new size cap.

Step 5: Verify the fix

journalctl --disk-usage

Should now show a value under 500MB. Also check disk space again: df -h /var/log/journal.

If the main fix didn't work

Maybe the disk is completely full — 100% used. Then journald can't even write config changes. In that case:

Free up space elsewhere first. Clean old packages: sudo apt autoclean (or dnf clean all).
Remove old log files manually: sudo journalctl --rotate then sudo journalctl --vacuum-size=200M. The --vacuum-size command removes the oldest logs to hit that size target.
If still stuck, stop journald, delete the journal folder contents, restart journald. That's nuclear but works. Do this only if you can't even get a shell prompt.

Prevention tip

Set SystemMaxUse on every new server you build. I add it to my Ansible playbook under the systemd-journald role. That way it never becomes a surprise. Also, set up a cron job (or a systemd timer) to run journalctl --vacuum-time=7d weekly. Keeps the directory lean without you thinking about it.

One more thing: if you use logrotate alongside journald (some setups do), make sure they don't fight. journald already rotates its own files. Don't let logrotate touch /var/log/journal — it'll cause duplicate work and possible corruption. I learned this the hard way on a CentOS 7 box where both tried to rotate the same files. Logs disappeared for a week.

Summary: edit journald.conf, restart daemon, verify disk usage. That's it. No data loss, no reboot, no drama.