EKS Node Group Stuck at Scaling — Fix for Failed Scale-Up/Down

Quick answer for advanced users

Check your launch template's IAM instance profile — it's probably missing the required EKS worker node policy. Also verify subnets have enough free IPs.

Context and why this happens

I've seen this many times on EKS clusters running Kubernetes 1.24+. You create a node group, it launches some nodes fine, then after a few weeks scaling stops working. You try to increase desired count in the console or via eksctl, but nothing happens. No errors in CloudTrail either. What's actually happening here is that the node group's launch template references an IAM role that doesn't have the AmazonEKSWorkerNodePolicy attached. Or the subnets in your VPC ran out of available IP addresses. AWS doesn't give you a clear error — it just silently fails. The other common cause: the launch template specifies a security group that doesn't allow outbound traffic to the EKS control plane. Your nodes spin up, can't register, and get terminated by the node group controller.

Fix steps

1. Check the node group's launch template. Go to EC2 → Launch Templates, find the one your node group uses. Click Actions → Modify template (create a new version). Scroll to Advanced details. Look at IAM instance profile. It must reference a role that has these policies: AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly.
2. If IAM profile is missing or wrong, create a new launch template version with the correct profile. Then update the node group to use this new version. You can do this in the EKS console: select node group, Actions → Update, launch template version.
3. Check subnet IP availability. Go to VPC → Subnets for the subnets your node group uses. Look at Available IPs. If any subnet has less than 3 free IPs, nodes will fail to launch. Create new subnets with more IPs and update the node group's subnet list.
4. Verify security group rules. The launch template's security group must allow outbound TCP 443 to the EKS control plane security group. Also allow inbound from the control plane on port 10250 and 10255. If you're unsure, use the default node security group that EKS creates.
5. Restart the node group scaling operation. After fixing the above, set desired size to current+1, wait 5 minutes, then set it back. This triggers a fresh scaling attempt.

Alternative fixes if main one fails

If your node group still won't scale after checking the above, try these:

• Check capacity reservations. If you have capacity reservations for an instance type in another AZ, nodes might launch in the wrong AZ and fail. Cancel unnecessary reservations.
• Delete and recreate the node group. I know it's painful, but sometimes the node group's internal state gets corrupted. Use eksctl or the console to delete and recreate it with the same launch template. Takes 10-15 minutes and usually works.
• Check service quotas. You might have hit a vCPU limit for the region. Go to Service Quotas, request increase for EC2 instances (m5.large or whatever you use).

Prevention tip

Before creating a new node group, always test the launch template by manually launching a single instance from it in the same subnets. If that instance shows up in the console and can curl https://your-cluster-endpoint:443, your template is good. Also set up a CloudWatch alarm on the node group's NodeGroupFailedToLaunch metric — it won't catch everything but it's better than nothing. And always keep at least 10 free IPs per subnet per node group. This saves you hours of debugging later.

One more thing: if you're using custom AMIs, make sure they have the EKS bootstrap script at /etc/eks/bootstrap.sh. I once wasted two days because my AMI was missing it.