AWS Network ACL Blocking Legitimate Traffic? Here's the Fix

I know it's maddening when you've double-checked everything, and traffic still gets dropped by a Network ACL. You've probably already looked at security groups—they're stateful and forgiving. But NACLs are stateless and brutal: one wrong rule and it's game over. Let's fix this.

The Fix: Check Your Ephemeral Ports and Rule Order

The most common mistake is forgetting that NACLs are stateless. That means if you allow inbound traffic on port 443 (HTTPS), you must also allow the corresponding outbound traffic for the response—and vice versa. And those responses often come from ephemeral (or dynamic) ports.

Here's what you do:

1. Identify the traffic direction: Is the traffic inbound to your subnet from the internet? Or outbound from your instances to somewhere else?
2. Check both inbound and outbound rules: If you're allowing inbound HTTP (port 80), you need an outbound rule that allows traffic from your instances back to the client—usually on ephemeral ports 1024–65535, with destination 0.0.0.0/0.
3. Rule order matters: NACL rules are evaluated from lowest number to highest. The first allow or deny that matches the traffic wins. So if you have a deny rule with a lower number than your allow rule, the deny fires first. Move your allow rules to lower numbers (e.g., 100 for allow, 200 for deny).
4. Check your source/destination: Are you using the right CIDR blocks? If your web server is in a public subnet and you're allowing inbound from 0.0.0.0/0 on port 443, that's fine. But if your outbound rules are too restrictive—like only allowing to specific IPs—responses to random clients will fail.

Here's a concrete example. I once helped a team whose NACL was blocking all HTTPS traffic to an ALB. Inbound rule #100 allowed HTTPS from anywhere. But outbound rule #110 allowed only HTTP to a specific internal server. The ALB's responses back to the client were using ephemeral ports and got denied by the implicit deny rule at the end (rule #*). Adding an outbound allow rule for ephemeral ports to 0.0.0.0/0 at rule #120 fixed it instantly.

Why This Works

NACLs are stateless—they don't track connections. Each packet is evaluated independently. Security groups are stateful: they automatically allow return traffic for an allowed inbound connection. With NACLs, you have to explicitly allow both directions. The ephemeral port range (1024–65535) is where clients pick random source ports for requests. If you don't allow outbound traffic from your server on those ports back to the client, the client never gets the response. And it's not just for web traffic—DNS, SSH, RDP, and most protocols use ephemeral ports on the client side.

Less Common Variations

1. Custom Protocol (non-TCP/UDP)

If you're using ICMP (ping) or a custom protocol like ESP for VPNs, you need rules for those protocols specifically. ICMP uses no ports—just type and code. In the AWS console, when adding a NACL rule, select "Custom Protocol" and specify ICMP (protocol 1) or ESP (protocol 50). I've seen people try to use TCP rules for ICMP—won't work. Use ICMP type 8 for echo request and type 0 for echo reply.

2. VPC Endpoints and Interface Endpoints

If you're using a VPC endpoint (like S3 or DynamoDB), your NACL must allow traffic to and from the endpoint's prefix list ID. Don't use the service's public CIDR—use the prefix list (e.g., pl-xxxxx). Same stateless rules apply: if your instance initiates a request to the endpoint, the NACL must allow inbound traffic from the endpoint's IP range on ephemeral ports.

3. Cross-Region Traffic or Transit Gateway

When traffic goes through a Transit Gateway or VPN, the NACLs on both sides must allow the traffic. If your on-premises network sends traffic to an EC2 instance in a subnet with a restrictive NACL, you need inbound rules for that traffic's source (the VPN endpoint IP or the on-premises CIDR) and outbound rules for the response. I've debugged a case where the outbound NACL only allowed to a single IP, but the VPN gateway's source IP was different for each packet—took a packet capture to see it.

Prevention: Build Your NACL Rules Like a Fortress

To avoid this pain in the future:

• Start with allow rules: Create a baseline NACL that allows only necessary traffic. For a web server, that might be: inbound HTTP/HTTPS from 0.0.0.0/0, outbound ephemeral ports to 0.0.0.0/0, and outbound SSH to your management IPs. Always allow ephemeral ports for responses.
• Number your rules in increments of 10 or 100: Leaves room to insert rules later. Don't use rule numbers 1–99 for something you might want to reorder.
• Test with VPC Flow Logs: Before applying to production, turn on VPC Flow Logs for the subnet. They'll show exactly which packets are being rejected and by which rule (look for "ACCEPT" vs "REJECT" in the log). I use this constantly to verify my NACL logic.
• Document the ephemeral port rule: Put a clear comment in your Terraform or CloudFormation template: "Allow outbound responses on ephemeral ports" so the next engineer doesn't delete it thinking it's too permissive.
• Use security groups first: Security groups are stateful and easier to manage for instance-level control. NACLs are best for subnet-level guardrails—like blocking known malicious IP ranges or ensuring certain ports are always closed at the subnet boundary.

That's it. Stateless ACLs require a mental shift, but once you get the hang of ephemeral ports and rule ordering, you'll never get caught again. Now go check those outbound rules.