Fix Kubernetes Etcd Quorum Lost in 5 Steps

Quick Answer

If you have one healthy etcd member left, promote it to a standalone cluster, then add new members. If all members are down, restore from a snapshot backup.

Why Etcd Loses Quorum

Etcd uses the Raft consensus protocol. It needs a majority of nodes (quorum) to agree on any change. A 3-node cluster needs 2 nodes. A 5-node cluster needs 3 nodes. When too many control plane nodes go down at once—say you rebooted all three masters for a kernel update and one didn't come back—you lose quorum. Etcd stops accepting writes. Pods stay running, but you can't deploy anything, and kubectl commands hang.

This happened to me when a power outage took out two out of three control plane nodes in a bare-metal cluster. The third node was still up, but it couldn't do anything alone. It was waiting for a friend that never showed up.

Step-by-Step Recovery

Step 1: Check the Damage

SSH into any control plane node that's still running. Run this command to check etcd membership:

ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list

You'll see which members are healthy (has leader: true) and which are unreachable. If you see only one member, you're in quorum lost territory.

Step 2: Stop etcd on All Nodes

On every control plane node, stop the etcd container (if using kubeadm):

docker stop $(docker ps | grep etcd | awk '{print $1}')

Or if you're using systemd directly:

systemctl stop etcd

This prevents any node from writing new data while you fix things.

Step 3: Promote the Healthy Member

On the node that has the most recent data (usually the one that was still responding), remove the old cluster configuration and start fresh as a single-node cluster. First, back up the existing data directory:

cp -r /var/lib/etcd /var/lib/etcd.backup.$(date +%s)

Then remove the member directory:

rm -rf /var/lib/etcd/member

Now update the etcd manifest file. For kubeadm, it's at /etc/kubernetes/manifests/etcd.yaml. Open it, and change the --initial-cluster-state flag from existing to new. Also change --initial-cluster to only include this node. For example:

--initial-cluster-state=new
--initial-cluster=node1=https://192.168.1.10:2380

If you don't see these flags in the manifest, add them under command:. Save the file.

Step 4: Start etcd and Verify

Kubeadm will automatically restart the etcd pod from the updated manifest. Wait 30 seconds, then check if it's healthy:

ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health

You should see 127.0.0.1:2379 is healthy: successfully committed proposal: took = X.Xms. If you see an error about auth or certs, double-check the paths in the manifest.

Step 5: Add Back the Other Nodes

Now that you have a single-node cluster, add the other control plane nodes back as new members. On the healthy node, get the peer URLs for the new members:

ETCDCTL_API=3 etcdctl member add node2 --peer-urls=https://192.168.1.11:2380

Repeat for each node. Then on each joining node, wipe the old data and restart etcd with the updated configuration pointing to the healthy node. The exact steps depend on whether you use kubeadm or a custom setup. For kubeadm, you can often just run kubeadm join again on the joining nodes—it will handle the etcd reconfiguration.

If This Doesn't Work: Restore from Snapshot

If none of your etcd nodes have a complete data set—say all three crashed hard and you don't have a healthy member—you need a snapshot backup. You do have one, right? If yes, restore it on a single node:

ETCDCTL_API=3 etcdctl snapshot restore /path/to/backup.db --name=node1 --initial-cluster=node1=https://192.168.1.10:2380 --initial-cluster-token=etcd-cluster --initial-advertise-peer-urls=https://192.168.1.10:2380 --data-dir=/var/lib/etcd/restored

Then point the etcd manifest to use the restored data directory. Don't forget to change the --data-dir flag in the manifest to /var/lib/etcd/restored. After that, follow Steps 4 and 5 above.

If you don't have a backup, you're in for a bad day. You'll need to spin up a new cluster from scratch and restore workloads from application backups. That's why I can't stress this enough: automate etcd snapshots to S3 or a local disk every hour.

Prevention: Keep Quorum Alive

The real fix is to not lose quorum in the first place. Here's what I do:

• Run an odd number of control plane nodes (3, 5, or 7). Never 2.
• Stagger your control plane updates. Reboot one, wait 5 minutes, then the next. Don't reboot all three at once.
• Use pod anti-affinity so etcd pods land on different physical hosts in cloud environments.
• Set up automated snapshots. Cron job + etcdctl snapshot save to an object store. Test the restore at least once a quarter.
• Monitor etcd member health with Prometheus and alert if any member goes down for more than 2 minutes.

If you follow these, you'll rarely see quorum lost. But when you do, the steps above will get you back online without rebuilding everything.