The Problem
You set termination protection on an EC2 instance, and it still gets terminated when your Auto Scaling group decides to scale in. I've seen this exact issue dozens of times. The culprit here is almost always a misunderstanding: termination protection only blocks manual termination from the console or CLI. It does nothing to stop Auto Scaling.
When an ASG scales in, it can terminate any instance that isn't explicitly protected at the ASG level. So if you've been relying on the EC2-level flag, you're fighting the wrong battle.
The Setup
You've got an Auto Scaling group running instances. You want to keep a specific instance alive no matter what the scaling policy says. Maybe it's a database, maybe it's a long-running job. You've already enabled DisableApiTermination on the instance via the EC2 console or CLI. But when the ASG decides to shed capacity, that instance gets a termination notice anyway.
Step 1: The Simple Fix (30 seconds) — Enable Scale-In Protection at the ASG Level
This is the first thing you should check. If you haven't set instance scale-in protection on the ASG, that's your problem.
You can do this via the console:
- Go to EC2 → Auto Scaling Groups.
- Select your ASG.
- Click the Instance management tab.
- Find the instance, select it, and click Actions → Set scale-in protection.
Or via CLI:
aws autoscaling set-instance-protection --instance-ids i-1234567890abcdef0 --auto-scaling-group-name my-asg --protected-from-scale-inThat's it. This tells the ASG to skip this instance when it's scaling in. It's the equivalent of a "do not kill me" flag for Auto Scaling.
Note: If you have a mixed instances policy or multiple instance types, this still works. Just apply it to the specific instance you care about.
One gotcha: If your ASG is already in the middle of a scale-in operation, the protection won't save an instance that's already been marked for termination. You might see a ScalingActivityInProgress error when you try to set protection. Wait for the activity to finish, then apply the protection.
Step 2: The Moderate Fix (5 minutes) — Use a Lifecycle Hook for Final Checks
Suppose you need more than just "don't kill it." Maybe you want to run some cleanup before the instance goes, or you need to drain connections. A lifecycle hook is your friend.
Lifecycle hooks pause the termination process at a specific point and let you run a script or send a signal before the instance actually dies. You can even cancel the termination if your script says so.
Here's how to add one:
- In the EC2 console, go to your ASG.
- Click the Instance lifecycle hooks tab.
- Add a hook for the
autoscaling:EC2_INSTANCE_TERMINATINGevent. - Set a timeout (e.g., 3600 seconds).
- Optionally specify a target Lambda function or SNS topic.
Now, when the ASG tries to terminate an instance (even one without scale-in protection), it'll wait. Your script can do whatever it needs, then call complete-lifecycle-action to let the termination proceed, or you can use the --action ABANDON flag to keep the instance if you change your mind.
Example CLI call:
aws autoscaling complete-lifecycle-action --lifecycle-hook-name my-hook --auto-scaling-group-name my-asg --lifecycle-action-result CONTINUE --instance-id i-1234567890abcdef0Be careful with hooks. If your script never completes the action, the instance will hang in a terminating state forever — or until the timeout expires. That can block scale-in entirely, which might be what you want, but it can also cause unexpected capacity issues.
Step 3: The Advanced Fix (15+ minutes) — Custom Termination Policy with Lambda
If you're dealing with a complex environment where different instances have different priorities, you might need a custom termination policy. This gives you control over which instances get terminated when scaling in.
Here's the idea: you configure the ASG to use a custom Lambda function as its termination policy. The Lambda receives a list of instances and returns the one it wants to terminate (or "None" to skip all).
First, create a Lambda function with a handler like this (Python example):
import json
def lambda_handler(event, context):
# event contains a list of instances
instances = event['Instances']
# Let's say we never want to terminate instances tagged as 'protected'
for instance in instances:
if 'protected' in instance.get('AvailabilityZone', ''): # just an example
continue
# Otherwise, terminate the one with the oldest launch time
if instance.get('LaunchTime'):
earliest = min(instances, key=lambda x: x['LaunchTime'])
return {'InstanceId': earliest['InstanceId']}
return {'InstanceId': None} # No instance to terminateThen attach it to the ASG:
aws autoscaling put-scaling-policy --auto-scaling-group-name my-asg --policy-name my-custom-policy --policy-type TargetTrackingScaling --target-tracking-configuration file://config.jsonActually, that's for a scaling policy. For termination policy, you set it in the ASG attributes:
aws autoscaling create-auto-scaling-group --auto-scaling-group-name my-asg --termination-policies "Custom::MyLambda" --launch-template LaunchTemplateName=my-templateYou'll also need to grant the Auto Scaling service permission to invoke your Lambda. This is getting into deep territory, but it's the most flexible solution.
But honestly? For most cases, Step 1 is enough. Step 2 is for when you need to do something on the way out. Step 3 is for when you have a complex, multi-tiered setup and you need surgical control.
Why Termination Protection Fails — The Real Reason
The EC2 DisableApiTermination flag works by denying any API call that would terminate the instance. Auto Scaling, however, doesn't call the TerminateInstances API. It uses its own internal mechanisms to shut down instances. So the flag is simply bypassed.
That's by design. AWS expects you to use the ASG's own protection mechanisms. The docs are clear: "To prevent an instance from being terminated during scale-in, use instance protection." But people miss that because they assume the EC2-level protection is universal.
What to Skip
Don't bother with IAM policies that restrict TerminateInstances for the ASG's role. It won't stop Auto Scaling because it uses a service-linked role that bypasses those restrictions (or it'll error out and cause more problems). Also, don't try to use CloudWatch alarms to detect scale-in and then re-enable protection — it's a race you'll lose.
Stick with the ASG-native solutions. They work, they're supported, and they don't fight the platform.
Common Scenario
You're running a web app with a spot instance that holds a cache. You set termination protection on it because you don't want it to die when the spot price spikes. But your ASG has a target tracking policy that scales in during off-hours. That spot instance gets terminated because the ASG doesn't care about your protection flag. Enable scale-in protection on that instance and you're golden.
One more thing: if you're using a launch template with InstanceMarketOptions for spot, remember that spot instances are always at risk of interruption. Scale-in protection only covers ASG-driven terminations, not spot reclamation. For that, you'd want capacity reservations or a different strategy.
Verification
After you apply scale-in protection, test it. Manually reduce your ASG's desired capacity by 1 and watch what happens. The protected instance should remain, and another one (if any) should be terminated. You can also check the ASG's activity history to see what it did.
aws autoscaling describe-scaling-activities --auto-scaling-group-name my-asgYou should see a message like "Instance i-xxx was protected from scale-in" if things are working right.
That's it. Stop fumbling with the EC2 console and use the right tool. Your future self will thank you.