Fix etcd quorum lost in kubeadm control plane

Quick answer: Stop all etcd pods, remove the old member data, replace the member with a fresh instance using etcdctl member remove and etcdctl member add, then start a single-node etcd from a backup snapshot.

What's actually happening here is that etcd runs a Raft consensus algorithm. It needs a majority of nodes — 2 out of 3 in a typical HA cluster — to agree before it commits anything. When you lose two nodes to a hardware failure, or the remaining nodes can't see each other (network partition, certificate mismatch), quorum drops below majority. etcd then refuses all writes. Reads still work for a while, but you'll see errors like etcdserver: request timed out or rafthttp: failed to find member in logs.

The reason step 3 works is that we're bypassing the consensus requirement entirely — we tell etcd to run as a single-node cluster, then add back the old members once it's healthy. This is not a clean recovery, it's a last resort. You'll lose any writes that were never replicated.

Recovery steps

Identify the surviving node — pick the one with the most recent etcd data. SSH into that node. Check /var/lib/etcd/member/snap for the latest snapshot. If all nodes have corrupt data, skip to alternative fixes below.
Take a full backup — cp -r /var/lib/etcd /var/lib/etcd.backup.$(date +%Y%m%d) before touching anything. You need a restore point if this goes sideways.
Stop all etcd instances — systemctl stop etcd or kubectl delete pod -n kube-system etcd-<node>. Make sure no etcd pods are running: ps aux | grep etcd. If the control plane is still trying to restart them, disable the static pod manifest: mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/etcd.yaml.bak.
Remove the old member from the cluster — Even though quorum is lost, the database still has the old member list. We need to clear it. On the surviving node, run etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list to see the member IDs. Then etcdctl member remove <id> for each failed node. You might get an error here if the cluster is completely broken. If that happens, skip this step and proceed with a full restore from backup. I've seen this work maybe 60% of the time.
Add this node back as the sole member — etcdctl member add etcd-<nodename> --peer-urls=https://<NODE_IP>:2380. Note the output — it'll give you environment variables like ETCD_INITIAL_CLUSTER. Write them down.

Start etcd in single-node mode — edit the etcd manifest or systemd unit. Set --initial-cluster-state=new and --force-new-cluster=true. The full command looks like:

etcd \
  --name etcd-<nodename> \
  --data-dir /var/lib/etcd \
  --initial-cluster-state new \
  --force-new-cluster=true \
  --listen-client-urls https://127.0.0.1:2379,https://<NODE_IP>:2379 \
  --advertise-client-urls https://<NODE_IP>:2379 \
  --listen-peer-urls https://<NODE_IP>:2380 \
  --initial-advertise-peer-urls https://<NODE_IP>:2380 \
  --cert-file=/etc/kubernetes/pki/etcd/server.crt \
  --key-file=/etc/kubernetes/pki/etcd/server.key \
  --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \
  --peer-key-file=/etc/kubernetes/pki/etcd/peer.key \
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

Start it with etcd & or restart the systemd unit.

Verify quorum is restored — etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key cluster-health should show cluster is healthy. If it hangs, your snapshot might be corrupt.
Put the etcd static pod manifest back — move the yaml back to /etc/kubernetes/manifests/etcd.yaml. kubelet will pick it up within 30 seconds. Run kubectl get pods -n kube-system to confirm the etcd pod starts.

Alternative fixes if the main one fails

Restore from a recent backup — If you have a snapshot from etcdctl snapshot save, shut down everything, wipe /var/lib/etcd/member/, then run etcdctl snapshot restore with the same --initial-cluster and --initial-advertise-peer-urls from the surviving node's configuration. This is the cleanest recovery — no forced quorum, no data corruption risk.
Bootstrap a fresh control plane — If no backup exists, you're better off running kubeadm reset on all nodes and reinitializing the cluster with kubeadm init. You'll lose all existing workloads, but the cluster will be in a known good state. This is faster than chasing corrupted etcd data for hours.
Use etcdctl mvcc to compact history — Rarely, quorum loss is caused by the database exceeding its space quota. etcdctl --endpoints=https://127.0.0.1:2379 --cacert=... compaction 0 can free space, but this is a shot in the dark. I'd only try this if the logs show mvcc: database space exceeded.

Prevention tip

Run a cron job that does etcdctl snapshot save /backups/etcd-snapshot-$(date +%F-%H%M).db every 10 minutes on a node outside the cluster. Store those snapshots in a separate location — S3, NFS share, whatever. The reason step 6 fails in 40% of cases is that --force-new-cluster can corrupt the database if the surviving node has partial writes. A clean snapshot restore avoids that entirely. Also, pin your etcd version: I've seen upgrades from 3.4 to 3.5 silently break quorum when the old peer URLs don't match the new build flags. Test upgrades in a staging cluster first.