Fix Kubernetes Node Not Ready - Runtime Stuck

You see a node stuck in NotReady state. The kubectl get nodes output shows it, and kubectl describe node says something like Node status is now: NodeHasSufficientMemory, NodeHasNoDiskPressure, NodeNotReady. The real trigger is usually one of two things: the container runtime (containerd or Docker) crashed and kubelet lost the connection, or the kubelet itself deadlocked after a kernel update or OOM event. I've seen this happen after a yum update that rebooted the kernel but not the services, or when a disk fills up and containerd can't write images.

What's actually happening here?

Kubelet relies on a CRI (Container Runtime Interface) gRPC socket. By default on modern Kubernetes (1.24+), it's /run/containerd/containerd.sock. When that socket becomes unresponsive — maybe from a containerd crash, a socket file corruption, or a kernel hang — kubelet can't create or check pods. After a few failed health checks (default is 40 seconds with 1-second interval), kubelet marks the node as NotReady and sets the NodeReady condition to False. The node stays that way until the runtime comes back.

The fix isn't to restart kubelet first. Kubelet will keep failing if the runtime is broken. You must fix the runtime, then bring kubelet back online.

The fix: restart the container runtime

SSH into the broken node. Use your jump box or direct SSH. You need root or sudo.
Check if containerd is running. Run:
```
systemctl status containerd
```
Look for Active: failed or Active: inactive (dead). If it's running but stuck, check the socket:
```
ls -la /run/containerd/containerd.sock
```
If the socket file is missing, containerd hasn't created it yet. If it's there but the service is hung, proceed.
Restart containerd hard. Don't use systemctl restart alone — that sometimes doesn't kill stuck processes. Do:
```
systemctl kill containerd
systemctl stop containerd
systemctl start containerd
```
Wait 10 seconds, then check status:
```
systemctl status containerd
```
It should show Active: active (running).
Verify the runtime responds. Use crictl to test:
```
crictl ps
```
If this returns a list of running containers (or an empty list), the socket works. If it errors with connection refused, the socket isn't up yet — wait 5 seconds and retry, or check containerd logs:
```
journalctl -u containerd --since "5 minutes ago"
```
Look for permission denied or file system errors.
Restart kubelet. Now that the runtime is healthy:
```
systemctl restart kubelet
```
Wait 30-60 seconds. Check the node status from your control plane:
```
kubectl get nodes
```
The node should flip to Ready within a minute. If it doesn't, check kubelet logs:
```
journalctl -u kubelet --since "2 minutes ago" | tail -50
```
Look for Successfully initialized node or Failed to connect to runtime.

If it still fails

Three things to check:

Disk space. Containerd needs room to pull images and write overlay mounts. Run df -h and docker system df (or crictl info). If /var/lib/containerd is above 90% full, garbage collection will fail. Clean old images: crictl rmi --prune or docker image prune -a if using Docker.
CRI socket mismatch. Kubelet might be configured for Docker but containerd is running. Check /var/lib/kubelet/kubeadm-flags.env and look for --container-runtime-endpoint. On Kubernetes 1.24+ with containerd, it should be unix:///run/containerd/containerd.sock. If you're still using Docker, change to unix:///var/run/dockershim.sock — but note dockershim is removed in 1.24+, so upgrade.
Kernel module issues. Sometimes after a kernel update, the overlay or br_netfilter modules aren't loaded. Run lsmod | grep overlay and lsmod | grep br_netfilter. If missing, load them:
```
modprobe overlay
modprobe br_netfilter
```
Then restart containerd and kubelet again.

I've also seen cases where the node's network interface dropped — ip link shows the interface down. That can cause kubelet to fail health checks even if the runtime is fine. Check ip a for the primary interface (like eth0 or ens192) — if it's down, bring it up with ip link set eth0 up.

One last thing: if you're on a cloud provider (AWS, GCP, Azure) and the node is part of a managed node group (like EKS managed nodes), don't fix it by hand. Delete the node from the cluster and let the auto-scaler or node group replace it. Hand-fixing managed nodes is a bad idea — the cloud controller will fight you.