Fixing STATUS_IPSEC_DOSP_MAX_ENTRIES (0xC0368004)

Most Common Cause: IPsec DoS Protection Cache Full

This error pops up when Windows IPsec’s denial-of-service protection has reached its maximum number of state entries. You’ll see this on servers or workstations handling lots of VPN connections — especially if you’ve got a remote access server or a site-to-site tunnel that keeps dropping and re-establishing.

The culprit here is almost always a flooded IKE (Internet Key Exchange) main mode security association cache. Windows limits this to 500 entries by default. Once you hit that, new IPsec negotiations get blocked and you get 0xC0368004 in the event log or as a netsh output.

Fix 1: Clear the IKE Security Association Cache

Open PowerShell as admin and run:

netsh advfirewall consec show rule name=all

That’ll list your connection security rules. Then clear the IKE main mode SAs:

netsh advfirewall consec delete rule name="all"

Wait — that nukes all rules. Instead, use this to clear only the cache:

Clear-NetIPsecMainModeSA -ErrorAction SilentlyContinue

That PowerShell cmdlet removes all stale IKE main mode SAs without touching your rules. Do this and try your VPN connection again. If it works, the problem was a full cache.

If you see the error return within hours, you’ve got a leak — old SAs aren’t timing out properly. Move to the next fix.

Second Most Common Cause: Default DoS Protection Limit Too Low

If clearing the cache only buys you a few hours, you need to raise the IPsec DoS protection entry limit. The default of 500 works for most small offices, but if you’ve got more than a few hundred concurrent VPN users or a busy site-to-site setup, you’ll hit it fast.

Fix 2: Increase the IPsec DoS Max Entries

Use PowerShell or regedit. PowerShell is easier:

Set-IPsecSetting -RemoteMachineTunnelLogging $false -IkeExeptionSeconds 0 -IpsecThroughputRateInKbps 0 -MaxRemoteMachineTunnelSAs 2000 -MaxRemoteUserTunnelSAs 2000

That bumps the main mode SA limit to 2000. Adjust the numbers based on your environment. I usually set it to double the expected peak concurrent connections.

If you prefer the registry route, go here:

HKLM\SYSTEM\CurrentControlSet\Services\PolicyAgent\Parameters

Create or modify these DWORD values:

• MaxNegotiationEntries – set to 2000 (decimal)
• MaxAuthFailEntries – set to 500 (decimal)

Reboot the service or the machine for changes to take effect:

Restart-Service PolicyAgent

I’ve seen this fix work on Server 2016, 2019, and Windows 10 20H2 and later. Don’t bother with the GUI firewall snap-in — it doesn’t expose these settings.

Third Most Common Cause: Misconfigured Firewall Rules Causing Half-Open Sessions

Sometimes the error isn’t about hitting a hard limit — it’s about rogue traffic keeping old SAs alive. If you have a firewall rule that’s too broad (like allowing any IP to initiate IPsec), external scanners or misconfigured peers can open hundreds of half-finished IKE sessions. Each one counts toward the 500 limit.

Fix 3: Tighten Connection Security Rules

First, identify which rule is causing the flood. Run:

netsh advfirewall consec show rule name=all verbose

Look for rules with Endpoint1 or Endpoint2 set to Any. That’s your problem. Change those to specific IP ranges or subnets.

Use the Windows Firewall with Advanced Security MMC (wf.msc). Edit each connection security rule and set the endpoints to your trusted networks only. For example, change from:

Endpoint1: Any
Endpoint2: Any

To:

Endpoint1: 192.168.1.0/24
Endpoint2: 10.0.0.0/8

After tightening, clear the IKE cache again (Fix 1) and monitor. If the error stops, you had a leaky rule.

One more thing — check for duplicate rules. I’ve seen admins accidentally create the same rule under both inbound and outbound, doubling the session count. Delete the duplicate.

Quick-Reference Summary Table

That’s it. Start with clearing the cache, bump the limit if needed, and lock down those firewall rules. You’ll have this error squashed in under 10 minutes.