Fix Kubernetes Pods Stuck in Terminating (Finalizer)

You kicked off a deploy at 11pm, a pod on node ip-10-0-3-117 got stuck in Terminating, and it's still there an hour later. You've tried kubectl delete pod three times. It returns instantly. The pod laughs at you. Sound familiar? I've been there, and it's one of the most infuriating Kubernetes states because nothing in the standard workflow tells you why it won't die.

This specifically happens when you're running custom controllers — cert-manager, Istio, an operator you wrote, or anything that registers a finalizer on a pod. It also shows up on nodes that went NotReady while a pod was terminating, and on clusters using StatefulSets with volumeClaimTemplates.

What a finalizer actually is

A finalizer is just a string in the pod's metadata.finalizers list. When you delete a pod, the API server sets deletionTimestamp, then waits. It won't remove the object from etcd until every finalizer in that list has been removed — usually by the controller that added it.

The deadlock part is simple. If the controller that's supposed to remove the finalizer is dead, uninstalled, or never reconciled the pod, that finalizer sits there forever. The kubelet can have already killed every container in the pod. Doesn't matter. The API object stays because the API server is doing exactly what it was told: wait for the finalizer to clear.

This tripped me up the first time because kubectl get pod shows Terminating and you assume the process is hanging. It isn't. The container is gone. The object in etcd is what's stuck.

Confirm the diagnosis

Get the full object and look at two fields:

kubectl get pod my-pod -n my-namespace -o json | jq '.metadata.finalizers, .metadata.deletionTimestamp'

If finalizers has entries and deletionTimestamp is non-null, you've got a finalizer deadlock. If finalizers is empty and it's still Terminating, the problem is lower — usually the kubelet on the node or the container runtime. Skip to the end for that case.

Check the node too:

kubectl get node ip-10-0-3-117

If the node is NotReady or SchedulingDisabled and the pod has been terminating longer than terminationGracePeriodSeconds, you're looking at the same symptom with a different root cause.

The fix

Try the normal path one more time, but watch the events. Run kubectl delete pod my-pod -n my-namespace --wait --timeout=60s and in another terminal run kubectl describe pod my-pod -n my-namespace. If you see nothing under Events, the delete is being swallowed by finalizers, not by a slow shutdown.
Identify which finalizer is the blocker. Look at the list from the jq output. Common ones are kubernetes.io/pvc-protection, foregroundDeletion, batch.kubernetes.io/job-tracking, and custom ones like istio.io/rev or cert-manager.io/issuer. The name usually tells you which controller owns it.
Remove the finalizer directly. This is the real fix. The containers are already dead, so there's nothing to clean up on the node. Patch the finalizers list to empty:
```
kubectl patch pod my-pod -n my-namespace -p '{"metadata":{"finalizers":null}}' --type=merge
```
The moment that patch lands, the API server sees no finalizers and removes the object. The pod disappears from kubectl get pods within a second or two.

If the pod has foregroundDeletion, you need to clear it explicitly. That finalizer is added by the garbage collector and won't clear if a dependent object is also stuck:

kubectl patch pod my-pod -n my-namespace -p '{"metadata":{"finalizers":["foregroundDeletion"]}}' --type=merge
kubectl delete pod my-pod -n my-namespace --grace-period=0 --force

Check for orphaned replicas. If the pod belongs to a Deployment or StatefulSet, the controller will usually spawn a replacement immediately. For a StatefulSet, the replacement won't come up until the old pod object is gone — which is exactly why clearing the finalizer matters here. Verify with:
```
kubectl get pods -n my-namespace -o wide | grep my-pod
```
Fix the controller that added the finalizer. If this happened once, it'll happen again. Find the controller with kubectl get events -n my-namespace --field-selector involvedObject.name=my-pod and kubectl logs -n my-namespace deploy/<controller> --since=1h. The controller is either crashed, doesn't have RBAC to update pods, or is stuck in a reconcile loop. That's the actual bug. The finalizer patch is just unblocking today.

If it's still stuck

Two scenarios are worth checking before you go nuclear on etcd.

The node is unreachable. If the node hosting the pod is NotReady, the kubelet can't confirm the pod was killed, so the API server holds the object. Either bring the node back, or if it's gone for good, remove it:

kubectl delete node ip-10-0-3-117

The node controller will then clean up pods that were bound to it. On managed clusters (EKS, GKE, AKS), the autoscaler usually handles this after about 20 minutes, but a stuck node in a bad AZ can hang around longer.

Volume detach is blocking. Pods using PVCs on AWS EBS, Azure Disk, or vSphere volumes can hang if the CSI driver won't detach. Check the VolumeAttachment objects:

kubectl get volumeattachment | grep my-pvc
kubectl describe volumeattachment <name>

If the attachment is stuck InProgress, the CSI controller pod on the control-plane side is the problem. Restart it, and the detach will complete. Then the pod will finally clear, usually without you needing to touch finalizers at all.

One last thing: kubectl delete pod --force --grace-period=0 on its own does not remove finalizers. People confuse the two constantly. It sends SIGKILL and skips the graceful shutdown, but the finalizer still holds the object. You need the patch. Do the patch first, then force-delete if you want, but if the containers are already dead the force flag is theater.