Container Pod Won't Schedule: Quick Fix to Deep Debug
Pod stuck pending? Start with taints and tolerations, then check resource requests, then dig into node conditions. Here's the real fix flow.
30-Second Fix: Check Taints and Tolerations
Nine times out of ten, a pod won't schedule because of taints and tolerations. Here's what's actually happening: your node has a taint that repels pods without the matching toleration. The scheduler sees this and moves on.
Run this command to see taints on all nodes:
kubectl describe nodes | grep -A5 Taints
If you see something like node.kubernetes.io/unreachable:NoSchedule, that's your problem. Now check the pod's tolerations:
kubectl get pod <pod-name> -o yaml | grep -A10 tolerations
No tolerations? Add them. The reason step 3 works is the scheduler can't override taints. It's a hard rule. Common real-world trigger: you moved a node to maintenance mode with kubectl cordon but forgot to remove the taint. Quick fix — remove the taint:
kubectl taint nodes <node-name> node.kubernetes.io/unreachable:NoSchedule-
That trailing dash removes the taint. Pod will schedule within seconds.
5-Minute Fix: Resource Requests and Quotas
If taints aren't the issue, look at resources. Pods need CPU and memory requests, not just limits. The scheduler checks if a node has enough allocatable resources for the sum of all pods' requests.
Check node capacity and allocatable:
kubectl describe node <node-name> | grep -A5 Capacity
kubectl describe node <node-name> | grep -A5 Allocatable
Allocatable is what matters. If your pod requests 2 CPUs but only 1.5 CPUs are allocatable, it won't fit. The scheduler doesn't overcommit unless you explicitly allow it with --enforce-node-allocatable flags.
Next, check if a ResourceQuota exists in the namespace:
kubectl get resourcequota -n <namespace>
If quotas are set, the entire namespace might be capped. Run:
kubectl describe resourcequota -n <namespace>
Look at pods count and requests.cpu — if they hit the limit, no more pods schedule. The real fix here is either increase the quota or reduce pod requests.
Also check if your pod has a PersistentVolumeClaim. If the PVC is pending, the pod stays pending too. Run:
kubectl get pvc -n <namespace>
If PVC shows Pending, the storage provisioner didn't create a volume. That's a different problem — check StorageClass and provisioner logs. But the scheduling failure itself is just a symptom.
15+ Minute Fix: Node Conditions and Scheduler Logs
If the above didn't help, dig into node conditions and scheduler logs. This is where most people stop and cry. Don't. It's straightforward once you know what to look for.
First, get the pod's scheduling events:
kubectl describe pod <pod-name>
Look at the Events section at the bottom. You'll see something like:
FailedScheduling 0/3 nodes are available: 1 Insufficient memory, 1 node(s) had taint, 1 node(s) didn't match pod affinity
That message tells you exactly why. Write it down. Each reason points to a different cause:
- Insufficient memory — resource issue from step 2, but double-check node memory pressure. Run
kubectl top nodesto see actual usage vs allocatable. - Taint — back to step 1, but maybe there's a
NoExecutetaint that evicts existing pods. Check withkubectl describe nodes | grep Taint. - Pod affinity — your pod has rules like
requiredDuringSchedulingIgnoredDuringExecutionthat limit where it can go. Review the pod yaml'saffinitysection. Common mistake: settingpodAntiAffinitythat conflicts withpodAffinity. Remove one.
If the events are vague, check the kube-scheduler logs directly. On a control-plane node with kubeadm, logs are at:
journalctl -u kube-scheduler -n 100
Or if using static pods:
kubectl logs -n kube-system kube-scheduler-<your-control-plane-node>
Look for lines with WARN or ERROR. The scheduler writes a message for every pod it can't schedule. Something like:
pod <pod-name>: node <node-name> didn't match node selector
That means your pod has a nodeSelector that doesn't match any node's labels. Check with:
kubectl get nodes --show-labels
Then compare to your pod's nodeSelector field. Fix the selector or add the label to a node.
One more thing: check if priority or preemption is causing issues. If your pod has low priority and higher-priority pods are waiting, your pod will stay pending. Run:
kubectl get priorityclass
If you see a default class with value 1000000000 and your pod uses class 0, preemption might evict your pod. The fix: assign a higher priority class. But be careful — don't set priority too high or it'll starve other pods.
Finally, if nothing works, delete the pod and recreate it. Sometimes the scheduler has a stale internal state. Run:
kubectl delete pod <pod-name>
kubectl apply -f pod.yaml
That forces the scheduler to re-evaluate all conditions. I've seen this fix weird intermittent failures where logs showed nothing wrong.
The whole point of this flow: start simple, avoid rabbit holes. Most scheduling failures are taints or resources. Only go to logs when you've ruled those out.
Was this solution helpful?