Fixing LVM metadata corruption on Linux servers

When this error hits

You reboot a Linux server after a power loss or kernel panic. And then you see it: Volume group "vg_data" not found or Can't open /dev/vg_data/lv_home. Maybe pvscan shows your physical disk but the volume group is missing. I had a client last month whose file server died during a UPS failure. The RAID was fine, the disks were fine, but LVM just couldn't see the volume group. That's metadata corruption.

Root cause – what went wrong

LVM stores metadata in two places: on the disk itself (usually at the start of each physical volume) and in a backup file at /etc/lvm/backup/. When something crashes mid-write – like a resize, a snapshot merge, or a failed pvcreate – the on-disk metadata gets scrambled. The kernel can't read it, so the volume group vanishes. The backup file is often still good, but only if you didn't overwrite it after the corruption.

Step-by-step fix

Do not reboot again. Do not run pvcreate or vgcreate. You'll wipe the metadata for good. Work from single-user mode or a live CD. I prefer SystemRescue, but any Linux live USB works.

1. Boot into single-user mode or a live environment. If you're on a remote server, you might need IPMI or a console. No network? Use a live USB physically.
2. Find your physical volumes. Run lvm pvs or pvscan. If you see a device listed but with unknown device or missing metadata, you're in the right place.
3. Check the backup files. Look in /etc/lvm/backup/. You'll see a file named after the volume group, like vg_data. Open it with cat /etc/lvm/backup/vg_data. If it has a metadata section with a creation_time and a list of physical volumes, you're in luck. If the file is empty or missing, skip to the alternative fix below.
4. Try a manual restore from backup. Use vgcfgrestore -f /etc/lvm/backup/vg_data vg_data. This writes the backup metadata back to the physical disks. Be sure you're restoring the right volume group name.
5. Activate the volume group. Run vgchange -ay vg_data. Check with lvs and mount to see if your logical volumes are back. If they mount, you're done.
6. If vgcfgrestore fails – it might say Can't open vg_data or Volume group not found. That means the backup has a different UUID or timestamp than the disks. In that case, you need to scan for old metadata on the disk itself.

Alternative fix – manual metadata recovery with pvck

If the backup is no good, you need to dig metadata straight off the disk. pvck is your tool. But it's dangerous and you must know exactly what you're doing.

1. Run pvck on each physical volume. Example: pvck /dev/sda1. It will show you metadata areas and labels. Look for lines like Found label at sector 1 and Found metadata at sector X.
2. Dump the metadata to a file. Use pvck --dump metadata /dev/sda1 > /tmp/metadata.txt. This gives you the raw metadata.
3. Read the metadata file. It's a text file with sections like physical_volumes, logical_volumes, and metadata_hash. Check if the UUIDs match what you expect. If you see entries for lv_root, lv_swap, etc., you're on the right path.
4. Restore the metadata manually. Use vgcfgrestore -f /tmp/metadata.txt vg_data. This is risky because the file's format must be exact. If it works, great. If not, you might need to reconstruct the metadata by hand – that's beyond this guide.
5. If pvck finds nothing – the metadata is completely gone. You can try pvcreate --uuid "original-uuid" --restorefile /dev/null /dev/sda1 to mark the disk as a PV without wiping data, but only if you have a backup of the UUID from /etc/lvm/archive/. Otherwise, you're looking at a full restore from backup.

What to check if it still fails

If the steps above don't work, here's what to look at next:

• Check the backup archive. Go to /etc/lvm/archive/. There are multiple backup files with timestamps. Try an older one. Sometimes the most recent backup is also corrupt.
• Check for duplicate UUIDs. Run blkid on all your disks. If two devices have the same UUID, LVM gets confused. Use pvchange -u /dev/sdb1 to assign a new UUID to one of them.
• Check the disk health. Run smartctl -a /dev/sda (or your device). If the disk has bad sectors, the metadata might be in a bad block. In that case, you need to replace the disk and restore from backup.
• Check if the volume group is exported. Run vgexport -a then vgimport -a to re-import all volume groups. Sometimes the metadata just needs a refresh.
• If nothing works, do a file-level recovery. Use fsck on the raw partition (without LVM) if you can. Or use testdisk to scan for lost partitions. It's ugly but it can save your data.

One last thing – always keep a current backup of your LVM metadata. Put vgcfgbackup -f /backup/lvm-%m-%d-%Y in your cron. It takes seconds and saves you hours of headache. Learned that one the hard way.