AKS Node Drain Stuck? 3 Fixes That Actually Work

First Thing to Try (30 Seconds): Check for Stuck Pods

Before you do anything else, run this command. It shows you exactly which pods are blocking the drain:

kubectl get pods --all-namespaces --field-selector spec.nodeName=YOUR_NODE_NAME --output wide

What you're looking for: pods stuck in Terminating or Pending status. These are the most common reason a drain hangs. After you run that, you'll see output like this:

NAMESPACE   NAME                   READY   STATUS        RESTARTS   AGE   NODE
default     my-app-7d8f9c5b6-9hk2m   0/1     Terminating   0          5m    aks-agentpool-12345678-vmss000001

If you see that—good, you found the problem. Now you can delete the pod directly:

kubectl delete pod my-app-7d8f9c5b6-9hk2m --namespace default --force --grace-period=0

After this command: the pod should disappear from the list. Then try your drain again with kubectl drain YOUR_NODE_NAME --ignore-daemonsets --delete-emptydir-data. If the drain finishes now, you're done. If it still hangs, move to the next section.

Moderate Fix (5 Minutes): Check Pod Disruption Budgets (PDBs)

Sometimes a single pod won't evict because a PodDisruptionBudget is blocking it. This is a common one. A PDB says "don't let too many pods of this app be down at once." If you have only 1 replica and a PDB that says minAvailable=1, the drain can't move that pod. Period.

Here's how to check for PDBs on your cluster:

kubectl get poddisruptionbudget --all-namespaces

If you see something like this:

NAMESPACE   NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
default     my-app-pdb   1               N/A               0                     2d

The ALLOWED DISRUPTIONS is 0. That means zero pods can go down at once. This is your blocker.

You have two options here:

Temporarily delete the PDB — risky but fast. Run kubectl delete poddisruptionbudget my-app-pdb --namespace default. Then run the drain again. Warning: this can cause downtime if the pod doesn't come back. Use only on non-critical workloads.
Scale up the deployment — safer. Run kubectl scale deployment my-app --replicas=2. That gives the PDB room to allow one pod to go down. Then drain works.

After either fix: run kubectl drain again. If it still fails, the next section is your last resort.

Advanced Fix (15+ Minutes): Force Drain with Pod Deletion

If the first two didn't work, you've got deeper issues. Usually it's a daemonset pod that won't stop, or a pod with a finalizer that's stuck. Here's the real fix:

First, get the full list of pods on the node again, but this time include all status details:

kubectl get pods --all-namespaces --field-selector spec.nodeName=YOUR_NODE_NAME -o wide | grep -v Running

Look for any pod that is not in Running status. Common culprits:

Pods with Init:Error or CrashLoopBackOff that never recover
Pods with a finalizer (like kubernetes.io/pv-protection) that block deletion
DaemonSet pods managed by an operator that ignores your drain

For each problematic pod, you need to force-remove it. But don't just delete blindly—check if there's a PVC attached first:

kubectl describe pod STUCK_POD_NAME --namespace YOUR_NAMESPACE | grep -i pvc

If you see a PVC, do not delete the pod unless you're willing to lose the data. Instead, you need to patch the pod to remove its finalizer:

kubectl patch pod STUCK_POD_NAME -p '{"metadata":{"finalizers":null}}' --namespace YOUR_NAMESPACE

After you run that, the pod should vanish. Repeat for all stuck pods.

Once all pods are gone, try the drain one more time:

kubectl drain YOUR_NODE_NAME --ignore-daemonsets --delete-emptydir-data --grace-period=30

If the drain still hangs after this, the node itself might be unreachable. In that case, you can skip draining and just mark it unschedulable, then let the cluster auto-replace it. Run:

kubectl cordon YOUR_NODE_NAME
# Then delete all pods manually
kubectl delete pod --all-namespaces --field-selector spec.nodeName=YOUR_NODE_NAME

After that: the node is cordoned and empty. You can safely delete the node in Azure:

az aks nodepool delete --resource-group YOUR_RG --cluster-name YOUR_CLUSTER --name YOUR_NODEPOOL_NAME

My opinion: Most people overthink this. 90% of drain failures are caused by a single stuck pod or a PDB. Try the 30-second fix first. If that doesn't work, check PDBs. The advanced fix is only needed when you have legacy apps with finalizers or bad PVCs. Don't jump to the big hammer first.