Distributed Storage Node Failure – Quick Fix for Ceph OSD Crashing

Let's get your OSD back online

I know seeing that OSD_DOWN status is annoying – especially when your cluster starts throwing warnings. Don't worry, we'll sort it out.

The quick fix – what to check first

Most of the time, a crashing OSD is caused by a bad disk or not enough memory. Here's what I do first:

1. SSH into the node with the failing OSD.
2. Run sudo ceph osd tree to see which OSD is down. Note the ID.
3. Check the OSD log: sudo tail -n 50 /var/log/ceph/ceph-osd.<ID>.log. Look for lines with "WARNING" or "ERROR". Common one is bluestore – failed to read.
4. Check disk health: sudo smartctl -a /dev/sdX for the disk that OSD uses. If you see Reallocated_Sector_Ct > 0 or Pending_Sector > 0, the disk is dying.
5. If the disk is fine, check memory: free -h. If you have less than 2GB free, increase RAM or move some OSDs to other nodes.

The real fix: if the disk is bad, replace it. Mark the OSD out, remove it, then add a new one.

sudo ceph osd out osd.<ID>
sudo ceph osd crush remove osd.<ID>
sudo ceph auth del osd.<ID>
sudo ceph osd rm osd.<ID>
# Then add a new OSD with a fresh disk

If it's memory, add more RAM or reduce the number of OSDs per node. Ceph needs about 1GB per OSD for normal operation.

Why this works

Ceph is picky about hardware. When a disk starts failing, the OSD can't read or write data fast enough. It times out and the cluster marks it down. Memory shortage causes the same issue – OSD processes get killed by the kernel's OOM killer. The fix above directly addresses both common triggers.

I've seen this on Ceph Octopus and Pacific releases, but it's the same on older versions. The smartctl test rarely lies – if it shows errors, replace that disk.

Less common variations

Sometimes the disk and memory are fine. Here's what else to check:

• Network issues – if the node has packet loss or high latency, OSDs may drop out. Run ping -c 100 <mon-ip> from the node. If you lose more than 1%, check your switch or cables.
• Kernel version – older kernels (like 3.x) have bugs with Ceph's bluestore. Check with uname -r. If it's below 4.15, update the kernel.
• Filesystem corruption – if the OSD uses XFS and the filesystem is damaged, the OSD crashes. Run sudo xfs_repair -n /dev/sdX to check. If it finds errors, unmount and repair.
• Clock skew – if the node's clock is more than 0.5 seconds off from the monitors, OSDs can fail. Use sudo timedatectl set-ntp true to sync.

How to prevent this

Once you fix the current issue, do these things to stop it happening again:

• Monitor disk health weekly with a script that runs smartctl and sends alerts if sectors go bad.
• Keep at least 2GB of free memory per OSD. For a node with 8 OSDs, that's 16GB minimum.
• Use enterprise SSDs or HDDs with good endurance. Consumer drives fail faster under Ceph load.
• Upgrade Ceph to the latest stable release. Each version fixes OSD crash bugs.
• Set up a cron job to scrub your pools regularly: ceph osd pool scrub <pool-name> once a week.

These steps have saved me from late-night emergencies more than once. Your cluster will stay healthy if you stay on top of disk health and memory.