When This Error Shows Up
You're running journalctl to check logs, and instead of seeing what happened, you get something like "journal corruption detected" or "failed to open journal file" or a whole bunch of binary garbage in the output. Maybe you just had a power outage, or your server crashed hard—like, a kernel panic or a hard reset. I had a client last month whose file server lost power right during a log rotation, and after booting back up, their logs were toast.
The error usually appears right after you try to read the journal with journalctl --verify or just journalctl itself. It can also show up in the system logs themselves: systemd-journald will spit out a message like "Corrupted journal file /var/log/journal/xxxxx/user-1000.journal, rotating." That's your system trying to save itself, but it doesn't always work.
Root Cause: What's Actually Wrong
Systemd-journald stores logs in binary journal files under /var/log/journal/ (or /run/log/journal/ if persistent storage isn't enabled). These files are structured like a database—they have headers, checksums, and indexed data. When a crash happens mid-write, that structure gets scrambled. The checksums don't match, or the file ends in the middle of an entry. systemd-journald tries to detect this and rotate the bad file away, but sometimes the damage is too deep, and it can't recover.
The real fix isn't about repairing the file—there's no fsck for journal files. You have to delete the corrupted files and let the journal build fresh ones. That's it. No magic tool, no elaborate recovery. Don't waste time trying to salvage them.
The Fix: Clear Corrupted Journal Files
Step 1: Stop systemd-journald
You can't safely delete journal files while the daemon is using them. Stop it first:
sudo systemctl stop systemd-journaldIf it's already stuck, you might need to kill it harder:
sudo systemctl kill -s SIGKILL systemd-journaldStep 2: Find and remove the bad files
The journal files live here:
/var/log/journal/Inside that, there's a machine-ID directory (a long hex string). That's where the .journal files sit. To see which ones are corrupt, run:
sudo journalctl --verifyIt'll list each file and say "PASS" or "FAIL" or "CORRUPTED". You want to delete the ones that fail. Or, if you're in a hurry (like I usually am), just wipe the whole directory and start fresh:
sudo rm -rf /var/log/journal/*Be careful: this deletes all past logs. If you need to keep some logs for compliance or debugging, only delete the specific corrupt files. But honestly, in most cases, you don't need old logs that much. Start clean.
Step 3: Restart systemd-journald
Once the files are gone, restart the service:
sudo systemctl start systemd-journaldIt will create a new machine-ID directory and fresh journal files automatically. Check if it's running:
sudo journalctl -n 10You should see recent entries—like the boot sequence and the restart itself.
What to Check If It Still Fails
If you still get corruption errors after doing this, a few things are probably going on:
1. Persistent storage isn't enabled
If you're running with /run/log/journal/ (volatile storage), the logs are in memory and might get corrupted differently. To check, run:
sudo journalctl --list-bootsIf it shows nothing or only the current boot, you're likely not saving logs to disk. Enable persistent storage:
sudo sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald2. Filesystem issues
Your disk might have bad blocks or filesystem corruption that's re-damaging the journal files. Run fsck on the partition holding /var/log. For example, if it's /dev/sda1:
sudo umount /var/log
sudo fsck -y /dev/sda1(You'll need to boot from a live USB if /var/log can't be unmounted easily.)
3. Low disk space
If the disk is full, systemd-journald can't write new files properly. Check with df -h. If it's full, delete some old logs or enlarge the partition.
4. A persistent hardware problem
If this keeps happening, like every week, it might be your RAM or storage dying. Had a client whose server kept corrupting journals—turned out their SSD was failing. Run smartctl -a /dev/sda and look for reallocated sectors or pending errors.
Bottom line: journal corruption is a pain, but the fix is simple: wipe and restart. Don't overthink it. Nine times out of ten, you'll be back up in five minutes.