You're staring at kubectl get nodes and one or more nodes show NotReady. Pods are pending, and your cluster is throwing fits. If you're on AWS EKS and using the default VPC CNI, there's a good chance you've hit IP address exhaustion. The CNI plugin hands out IPs from the node's elastic network interfaces (ENIs), and when it runs out, new pods can't get an IP. The node's kubelet can't reach the API server, so it flips to NotReady.
What's actually happening here is the CNI plugin tries to allocate an IP for the node's own networking, fails, and the node loses its connection. This isn't a Kubernetes bug; it's a capacity issue. The fix depends on how deep you want to go. Start with the 30-second band-aid, then move to the 5-minute tweak, and if you're still stuck, the 15-minute advanced fix.
30-Second Fix: Restart the aws-node Pods
Sometimes the CNI plugin just gets stuck in a bad state. A quick restart of the aws-node DaemonSet can free up leaked IPs and get your node back. It's not a permanent fix, but it buys you time.
kubectl rollout restart daemonset aws-node -n kube-system
That's it. Wait 30 seconds and check node status again. If the node goes Ready, you're done for now. But if it flips back to NotReady within minutes, you've got a deeper problem. The reason this works: restarting the CNI plugin clears its internal IP allocation cache and forces it to re-scan available IPs on the ENI. Leaked IPs from terminated pods get released.
Real-world trigger: This often happens after a sudden spike in pod creation, like a deployment scaling up too fast. The CNI plugin can't keep up, allocates IPs aggressively, and leaves some dangling when pods die.
5-Minute Fix: Increase ENI Limits or Free Up IPs
If the restart didn't stick, you're genuinely out of IPs. Each EC2 instance type has a maximum number of ENIs and IPs per ENI. The CNI plugin uses these to determine how many pods the node can support. If you're maxed out, new pods can't get IPs.
First, check how many IPs are available on the node's ENIs. SSH into the node (or use SSM) and run:
ip addr show | grep -c "inet "
That gives you a rough count. Better yet, check the CNI plugin logs:
kubectl logs -n kube-system -l k8s-app=aws-node --tail=100
Look for messages like failed to assign an IP address to pod or no available IP addresses. That confirms exhaustion.
The quickest fix is to increase the maximum pods per node by adjusting the ENI_CONFIG or using the WARM_IP_TARGET setting. But that requires a restart of the CNI plugin. Instead, try freeing up IPs by deleting unused pods or scaling down deployments. If you have pods stuck in Terminating, force delete them:
kubectl delete pod <pod-name> --grace-period=0 --force
That releases their IPs back to the CNI pool. You can also check for orphaned IPs using the CNI plugin's metrics if you have Prometheus set up. Look for awscni_assigned_ip_addresses vs awscni_total_ip_addresses.
If you're still stuck, you can temporarily increase the number of IPs by attaching a secondary ENI to the instance. But that's a manual hack. The real fix is to either use a larger instance type or enable prefix delegation.
15-Minute Fix: Enable Prefix Delegation or Switch to a Different CNI
Prefix delegation is the proper solution for IP exhaustion on EKS. Instead of allocating individual IPs, the CNI plugin assigns /28 prefixes (16 IPs) to ENIs. This dramatically increases the pod density per node. It's supported on Nitro-based instances and requires Kubernetes 1.21+ and CNI plugin version 1.9.0+.
To enable it, you need to update the aws-node DaemonSet to set ENABLE_PREFIX_DELEGATION to true.
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
Then restart the CNI plugin:
kubectl rollout restart daemonset aws-node -n kube-system
Why this works: The CNI plugin now requests a /28 prefix from the VPC, giving you 16 IPs at once instead of one by one. This reduces the chance of exhaustion and speeds up pod scheduling. Note that this requires the node's instance type to support enough ENIs and IPs. Check the AWS ENI documentation for limits.
If prefix delegation isn't an option (older instance types, for example), consider switching to a different CNI like Calico or Cilium. They manage IPs differently and can use overlay networking to avoid VPC IP exhaustion entirely. But that's a bigger architectural change.
One more thing: if you're using custom networking with ENIConfig, make sure your subnet has enough free IPs. A subnet with a /24 CIDR gives you 251 usable IPs, but if you have hundreds of pods, you'll run out. Create a larger subnet or add more subnets to the ENIConfig.
Preventing Future Exhaustion
Monitor your IP usage. Set up CloudWatch alarms for AvailableIpAddressCount on your subnets and ENIs. Use the CNI plugin's metrics to track assigned vs total IPs. And avoid using t3 or t2 instances for high-density pods; they have low ENI limits. Stick with m5, c5, or r5 and newer generations.
Also, consider enabling WARM_IP_TARGET and MINIMUM_IP_TARGET to control how many IPs the CNI plugin pre-allocates. Setting WARM_IP_TARGET=2 keeps a small buffer without hoarding IPs.
kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=2
That's the whole flow. Start with the restart, then free IPs, then go for prefix delegation. Each step buys you more time and gets you closer to a stable cluster.