IPsec DOS Throttle Drop 0xC0360009 Fix

When This Error Hits

You're managing a Windows Server 2019 or 2022 box that handles a lot of VPN connections—maybe a Remote Access server or a site-to-site IPsec tunnel. Suddenly, remote clients start failing to connect. Event ID 4654 pops up in Security logs with 0xC0360009 (STATUS_IPSEC_THROTTLE_DROP). The packet gets dropped before the IPsec Security Association (SA) completes.

I've seen this most often after a spike in connection attempts—like when 50 users try to reconnect after a network outage, or a misconfigured client keeps retrying every second. The throttle is designed to protect the server from a real DOS attack, but it doesn't know the difference between an attack and a legitimate rush of users.

Root Cause: The IPsec DOS Throttle

Windows IPsec has a built-in DOS protection mechanism. It tracks how many negotiation requests (IKE main mode or quick mode) hit the server per second. If the rate exceeds a default threshold—usually around 25 requests per second per source IP—the system starts dropping packets. This isn't a protocol error; it's a deliberate shield.

The problem? The default threshold is too low for busy servers. Microsoft set it conservatively to avoid CPU exhaustion, but modern hardware can handle way more. The fix is either to increase the limit or disable the throttle entirely (if you have another DOS solution in place).

How to Fix It

You've got two paths: adjust via the registry (persistent) or use netsh (for testing). I recommend the registry approach for production servers—it survives reboots.

Option 1: Increase the DOS Throttle via Registry

1. Open Registry Editor as Administrator (regedit).
2. Navigate to:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\Oakley
3. If Oakley doesn't exist, create a new Key under PolicyAgent.
4. Add a DWORD (32-bit) value named MaxNegotiationAttemptsPerSec.
5. Set it to 100 (decimal). That's four times the default. For really busy servers, go up to 200.
6. Reboot or restart the IPsec Policy Agent service: net stop PolicyAgent && net start PolicyAgent.

Warning: Don't set it to 0 (which disables the throttle) unless you have a dedicated firewall handling DOS. I've done it in labs—fine. In production, you're asking for trouble if a real attack hits.

Option 2: Quick Test with Netsh

If you want to test before committing to a registry change (or can't reboot right now), use netsh from an admin command prompt:

netsh ipsec dynamic set config property=MaxNegotiationAttemptsPerSec value=100

This change is immediate but doesn't survive a reboot. Use it to verify that the error stops. If it does, go with the registry method.

What If It Still Fails?

If the error persists after raising the throttle:

• Check for a second throttle: There's also MaxAuthenticatedNegotiationAttemptsPerSec—same location in the registry. This one limits authenticated IKE attempts (phase 2). Set it to the same value as the first.
• Look for IPsec policy mismatch: Sometimes clients send different authentication methods than the server expects. Run netsh ipsec static show policy name="YourPolicyName" and compare with client config.
• CPU saturation: If your server is pegged at 100% CPU, even raising the throttle won't help. The throttle is there for a reason—the server can't handle the load. Add more cores or offload IPsec to a dedicated device.
• Windows Firewall logging: Enable dropped packet logging (netsh advfirewall set currentprofile logging filename %systemroot%\system32\LogFiles\Firewall\pfirewall.log) to see if another rule blocks the traffic before IPsec gets a chance.

That last one tripped me up on a Windows 2016 box last year—a rogue firewall rule was dropping IKE UDP port 500 before IPsec even saw it. Took me two hours to find because the event log only showed the throttle error, not the firewall drop.

One more thing: check your certificate revocation. If you're using certificate-based authentication, the throttle can fire if the server can't reach the CRL distribution point and keeps retrying. Speed up CRL checking or use a local CRL cache.

You've got this. Raise that throttle, double-check the firewall, and your IPsec should start flowing again.