ZFS Pool Degraded: Missing Device Fix (3 Steps)

1. Quick Fix: Rescan and Reconnect (30 seconds)

What's actually happening here is the OS lost track of the drive's device path (like /dev/sda or /dev/disk/by-id/...). This often happens after a reboot, a loose cable, or a brief power glitch. The drive itself is probably fine.

1. Run sudo zpool status. Look for a line like FAULTED or UNAVAIL — note the pool name and device path.
2. Rescan the SCSI/SATA bus to force the kernel to re-detect the drive:

sudo echo "- - -" > /sys/class/scsi_host/host0/scan
sudo echo "- - -" > /sys/class/scsi_host/host1/scan
# Repeat for host2, host3 as needed — check `ls /sys/class/scsi_host/`

3. Check dmesg | tail -20 — you should see something like [sdX] Attached SCSI disk.
4. Now run sudo zpool online poolname /dev/disk/by-id/old-drive-id. Replace poolname and the path with what you saw in step 1.

The reason step 3 works is ZFS remembers the drive's GUID (a unique identifier), not just the device path. If the same GUID shows up again, ZFS can bring it back online. If this fails, move to the next step.

2. Moderate Fix: Replace Drive in Same Slot (5 minutes)

If the rescan didn't help, the drive might have changed its device path. This is super common when you swap SATA ports or add/remove other drives. The disk is still there, but ZFS can't find it at the old address.

1. Find the drive's serial number from your records (you did label your drives, right?) or check sudo lshw -class disk or sudo smartctl -a /dev/sdX for each visible drive.
2. Once you identify the correct disk, run:

sudo zpool replace poolname old-device-id /dev/disk/by-id/new-device-id

Replace old-device-id with the faulted path from zpool status and new-device-id with the actual working disk's ID (e.g., ata-WDC_WD40EFZX-68AWUN0_XXXX).

Important: never use plain /dev/sdX — those numbers change across reboots. Always use the /dev/disk/by-id/ or /dev/disk/by-path/ paths. The reason is ZFS stores the device path in its config. If you use /dev/sdb today, tomorrow it might be /dev/sdc, and your pool breaks again.

After the replace, ZFS will start resilvering (rebuilding redundancy). Check progress with zpool status. This might take hours for large drives, but the pool is usable during resilvering.

3. Advanced Fix: Full Drive Replacement (15+ minutes)

If the drive is truly dead (not detected by smartctl at all, clicking sounds, or won't spin up), you need a new physical drive. This is the nuclear option — only do this if the first two steps failed and you're sure the drive is gone.

1. Power down, swap the bad drive with a new one (same or larger capacity). Boot up.
2. Check the new drive is visible: lsblk | grep sdX (replace sdX with what you see).
3. Partition the new drive if needed — ZFS works fine on raw disks, but some setups use GPT partitions. I skip this unless the other drives have partitions.
4. Run the replace command but with the faulted device ID from step 1:

sudo zpool replace poolname old-faulted-id /dev/disk/by-id/new-disk-id

5. Resilvering starts automatically. You can check with zpool status — expect a line like resilver in progress.

A note on pool layout: If your pool is a single-disk vdev (like a mirror or RAIDZ), losing one drive means the pool is degraded but still functional for reads. If you lose too many drives in parity vdevs, the pool can fail completely. Don't let it get that far.

Why Did This Happen? Common Causes

• Loose cable — SATA cables wiggle loose over time. Reseat both ends.
• Power supply issues — A dying PSU can drop drives randomly. Check voltages in BIOS or with sensors.
• Backplane problems — On servers, hot-swap backplanes have bad slots. Move the drive to another slot and see if it shows up.
• Drive firmware bugs — Some WD Red drives (EARS/EFAX) have issues with TLER and drop off the bus. Check your drive's firmware version against known issues.
• Linux kernel changes — After a kernel update, device numbers can shift. That's why we use /dev/disk/by-id.

Prevention Tips

• Always label drives with their serial number and slot position. Use a label maker, not a Sharpie.
• Use /dev/disk/by-id/ for all your ZFS pools from the start. You can convert an existing pool by exporting and reimporting with the correct paths:

sudo zpool export poolname
sudo zpool import -d /dev/disk/by-id/ poolname

• Set up email alerts in ZFS for pool events. On FreeNAS/TrueNAS, this is under System > Email. For Linux, use zpool events and a cron job.
• Check your cables every 6 months. A simple reseat prevents 80% of these dropped-drive issues.

Final word: If none of these steps work, you might have a controller or backplane issue. Boot a live Linux USB and see if the drive appears there. If not, the drive is dead. Replace it and move on.