Fix Systemd Journal Storage Corrupted

When This Error Shows Up

You're running journalctl and get nothing but an error like "Failed to open journal file" or "No such file or directory" referring to a file under /var/log/journal/. Or maybe systemctl status shows a service is failing but the log is blank. I've seen this happen after a sudden power loss on a server running Ubuntu 20.04, a forced shutdown of a Debian 11 VM, or even after a disk fills up and the journal daemon can't write properly. One client last week had a Raspberry Pi running 24/7 that crashed overnight, and next morning journalctl came back completely empty.

Root Cause – Plain English

The systemd journal stores logs in binary files under /var/log/journal/ (if persistent logging is enabled) or in memory (/run/log/journal/). These files are structured databases, not plain text. When the system crashes or the journal daemon gets killed mid-write, the file header can get corrupted. This makes the whole file unreadable. The journal daemon then refuses to touch the broken file, and new logs can't be written either. It's like a database table that got a corrupted index – reads fail, writes fail. You don't need to reinstall the OS or even reboot. Just clear the bad file and let systemd rebuild it.

The Fix – Step by Step

This fix assumes you want persistent logging (logs survive reboot). If you only use volatile logging, skip to step 5. But I always tell clients: set up persistent logging – it's saved me more times than I can count.

Step 1 – Stop systemd-journald

You can't delete or modify journal files while the daemon holds them open.

sudo systemctl stop systemd-journald

Step 2 – Locate the corrupted journal files

Check /var/log/journal/ – that's the persistent storage directory. There will be a subdirectory named after your machine's ID (a 128-bit hex string). Inside that, you'll see system.journal and possibly user-1000.journal files.

ls -la /var/log/journal/$(cat /etc/machine-id)/

Step 3 – Remove the corrupted files

Don't worry – systemd will recreate them automatically. If you have valuable logs you need to recover, see the note below first.

sudo rm -f /var/log/journal/$(cat /etc/machine-id)/*

Step 4 – Start systemd-journald

The daemon will create fresh journal files on startup.

sudo systemctl start systemd-journald

Step 5 – Verify it works

Check that journalctl can now read logs:

journalctl -n 10

You should see recent entries. If you still see an error, check the daemon's own logs:

journalctl -u systemd-journald -n 20

What If You Still Get Errors?

A few things can go wrong:

Permissions issue – The /var/log/journal/ directory must be owned by root:systemd-journal with permissions 2755. Fix it: sudo chown root:systemd-journal /var/log/journal && sudo chmod 2755 /var/log/journal.
Disk full – If the partition holding /var/log is full, systemd-journald can't write. Free up space or set SystemMaxUse=500M in /etc/systemd/journald.conf.
Persistent logging not enabled – Check grep -i 'storage' /etc/systemd/journald.conf. If it says Storage=volatile or is commented out, set Storage=persistent and restart the daemon. Without persistent storage, logs vanish on reboot – and you'll see this problem every time.
Machine ID mismatch – Rare, but the subdirectory name might not match /etc/machine-id. Run cat /etc/machine-id and ls /var/log/journal/ – if they differ, move or rename the folder to match.

If none of that works, you can force a full vacuum of the journal – this nukes everything, but it's a nuclear option:

sudo journalctl --flush --rotate --rotate --vacuum-time=1s

Then restart journald. I've only had to use that once, on a server that had been running for 3 years without log rotation. It worked.

Pro tip: Before deleting, you can sometimes recover logs from a corrupted file by running journalctl --file /var/log/journal/XXX/system.journal – if the file is partially readable, you'll get some output. Pipe it to a text file before deleting. Saved me once on a client's production database server.

That's it. You don't need to reboot, you don't lose important logs (unless you skipped the recovery tip), and you're back in business in under five minutes. This is one of those Linux problems that looks scary but has a dead simple fix – as long as you know where the cleanup switch is.