Fix RDS Multi-AZ Failover Not Triggered

When This Error Happens

You're running an AWS RDS instance with Multi-AZ enabled. An availability zone (AZ) goes down — maybe a power failure or network issue in us-east-1a. You check CloudWatch, and your instance is still showing as 'available' but queries start timing out. The failover to the standby in us-east-1b never triggers, or it takes so long your app shows errors. This happens specifically when the primary is still 'running' but unreachable from your application, or when AZ degradation is partial (e.g., packet loss but not full outage).

Root Cause

AWS RDS Multi-AZ failover is not automatic in the way most people think. The standby instance sits idle — it doesn't accept traffic until AWS detects the primary is truly dead. The detection mechanism relies on:

Health checks at the hypervisor level
DNS propagation (which can lag)
Your application's connection timeout behavior

What's actually happening here is the primary AZ might be partially degraded — network packets still flow but with high latency or random drops. AWS's health check might still see the instance as 'reachable,' so no failover triggers. Meanwhile, your app sees timeouts. The fix isn't on AWS side — it's in your app and DNS settings.

The Fix (Step by Step)

Step 1: Check DNS TTL on the RDS CNAME

When failover happens, AWS updates the DNS record to point to the standby's IP. But if your app caches that DNS entry longer than 5 seconds, you'll keep hitting the dead primary. Set the TTL on your application's DNS resolver (not the RDS console) to 5 seconds or lower.

# In production, use a low TTL in Route53 or your app's DNS cache
# Example: nginx resolver cache config
resolver 127.0.0.53 valid=5s;

Step 2: Set Application Connection Timeout Below 30 Seconds

AWS's failover detection takes about 60-120 seconds by default. Your app should timeout way before that. Set your database connection timeout to 5 seconds max. If your app hangs for 30 seconds, it'll look like failover failed when really it just hadn't kicked in yet.

# Python (psycopg2)
import psycopg2
conn = psycopg2.connect(
    connect_timeout=5,
    host='your-rds-endpoint',
    ...
)

Step 3: Monitor AWS Health Dashboard and RDS Events

Don't rely on CloudWatch alone. Subscribe to AWS Personal Health Dashboard events for your RDS instance. These get pushed faster than standard CloudWatch metrics. Also enable RDS event subscriptions in SNS — they send messages when failover starts or ends.

# AWS CLI to subscribe to RDS events
aws rds create-event-subscription \
    --subscription-name my-failover-alert \
    --sns-topic-arn arn:aws:sns:us-east-1:123456789:my-topic \
    --source-type db-instance \
    --event-categories 'failover' 'failure'

Step 4: Enable Multi-AZ Failover in Read Replicas (If Used)

If you're using RDS read replicas, they don't automatically promote during a failover. You need to manually promote a read replica. AWS added Multi-AZ for replicas in 2023 — make sure it's enabled on each replica separately.

Still Not Working? Check These

Is the standby in a different AZ? Sounds dumb, but I've seen people create a Multi-AZ instance where both copies ended up in the same AZ due to resource constraints. Verify in RDS console under 'Configuration' → 'Multi-AZ' shows two distinct AZs.
Is your VPC routing correct? If the standby is in a private subnet without a NAT gateway or proper route tables, the failover happens but traffic can't reach it. Check route tables for both AZ subnets.
Are you using an Application Load Balancer? If you front RDS with an ALB (not common, but I've seen it), the ALB keeps connections alive to the old primary. Remove the ALB — it adds a layer that blocks failover detection.
Is this a test? Simulate failover manually: Use aws rds reboot-db-instance --db-instance-identifier my-db --force-failover. If that doesn't work, something is completely broken — open a support case with AWS.

One more thing: Multi-AZ failover is not instant. Even with DNS TTL at 5 seconds and app timeouts at 5 seconds, you'll see a ~10-15 second gap where queries fail. That's normal. Plan for it with retry logic in your app.