Proxmox Backup Checksum Mismatch – Quick Fix

You've run a verify on a Proxmox backup and got that red 'checksum mismatch' error. It's frustrating when the backup tool says the data's corrupt, especially if you're in a hurry. Let's fix this without panicking.

The Fix: Delete and Recreate the Index File

The real culprit in 90% of cases is a corrupted index file, not the actual backup data. Proxmox creates a .index file alongside your .vma or .vma.zst backup. If that index gets written incompletely (power glitch, file system hiccup, or a previous failed backup), verification fails even if the data blob is fine.

Here's what to do:

1. SSH into your Proxmox server. Use a root shell or a user with sudo access.
2. Go to the backup directory that's failing. For a standard PBS store, that's usually under /mnt/datastore/ or wherever you mounted it. Example:
   cd /mnt/datastore/backup-server/ct/101/2023-10-15T12:00:00Z
3. List the files to see the index and data file:
   ls -la
   You should see something like backup-ct-101-2023_10_15-12_00_00.vma.zst and backup-ct-101-2023_10_15-12_00_00.vma.zst.index.
4. Delete the index file only. Don't touch the data file yet.
   rm backup-ct-101-2023_10_15-12_00_00.vma.zst.index
5. Recreate the index using the Proxmox backup tool:
   proxmox-backup-manager verify-job run – this won't work because it's for jobs. Instead, use the direct verify command on the backup snapshot:
   proxmox-backup-client verify --cipher-encrypted if encrypted, or just proxmox-backup-client verify for unencrypted.
   But the fastest way is to run a manual verify through the web interface. Go to Datastore > your store > select the backup > click 'Verify'. Proxmox will regenerate the index from the data file.
6. Wait for the verify to finish. It should now pass. After completion, you'll see a fresh .index file that matches the checksum.

What you should expect: After step 5, the verification will start and might take a while depending on backup size. When it finishes, the status changes from 'failed' to 'OK'. The index file will be slightly smaller or larger than the original if the original was corrupted.

Why This Works

The index file isn't a backup of your data – it's a metadata map. It stores chunks and their hashes. A corrupted index means Proxmox reads a wrong hash from the index, compares it to the data's actual hash, and screams 'checksum mismatch!'. Deleting the index forces Proxmox to rebuild it from scratch, scanning the data file and computing correct hashes. The data file itself is almost always intact because ZFS, ext4, or Btrfs usually catch block-level corruption. The index gets written in a single atomic write, but if that write is interrupted, you get partial data.

One time I saw this on a backup server after a UPS failure. The index was 2 bytes short. Deleted it, re-ran verify, and it passed. The client's data was perfectly fine.

Less Common Variations of the Same Issue

1. Encrypted Backups

If you use client-side encryption, the verify process needs the encryption key. If the index is corrupted and encrypted, you can't just delete and re-verify – because Proxmox won't be able to decrypt the data to rebuild the index. In that case:
- Restore the key from your key file or key server.
- Then run proxmox-backup-client verify --keyfile /path/to/key.
If that fails, you may need to restore the backup to a temporary location and re-encrypt it. That's rare, but it happens.

2. Partial Backup from a Failed Job

Sometimes a backup job crashes mid-way, leaving a partial data file and a partial index. The index might have hashes for chunks that were never written. Verifying will fail on the missing chunks. Fix: Delete both the data and index files, then re-run the backup job from scratch. You can't salvage a partial backup – it's not index corruption, it's incomplete data.

3. File System Corruption on the Datastore

If multiple backups fail with checksum mismatches, you might have file system corruption on the backup storage. Check with zpool scrub (ZFS), btrfs scrub (Btrfs), or fsck (ext4). If you find errors, the data file itself could be corrupted. In that case, you'll need to delete that backup and restore from an older verified backup or from the source VM/CT directly.

4. Backup Set References

Proxmox Backup Server has a 'backup set' concept. If you delete a backup that's part of a verification group, the index for that set might still reference it. You'll get a 'checksum mismatch' for the set, not the individual backup. Fix: Remove the orphaned backup from the set via the web interface, then re-run verify.

Prevention

Here's how to stop this from happening again:

• Use a UPS. Power glitches during backup writes corrupt indexes. A cheap UPS on the backup server stops this cold.
• Run regular file system checks. Weekly scrubs on ZFS or Btrfs catch corruption early. Schedule them in cron.
• Set up verification schedules. In Proxmox Backup Server's web UI, create a verification job that runs weekly. It'll catch index corruption before you need that backup.
• Don't rely on a single backup. Keep 3-4 days of backups. If one gets corrupted, you can restore from an older one while you fix the new one.
• Monitor backup logs. Use the Proxmox log viewer or set up email notifications for verification failures. Don't wait until you need to restore to find out a backup is bad.

The bottom line: checksum mismatch is usually a broken index, not a broken backup. Delete the index, re-verify, and you're back in business. Keep those indexes clean and your recovery plan solid.