Fix IPsec IKE Invalid Header Error 0x36E0

When this hits

You're trying to connect a Windows 10 or Windows Server 2019 machine to a remote VPN gateway — maybe a Cisco ASA or a strongSwan server. Connection starts, then fails after a few seconds. You check the Event Viewer under Applications and Services Logs / Microsoft / Windows / RasClient / Operational, and there it is: ERROR_IPSEC_IKE_INVALID_HEADER (0X00003600). The event text says something like “IKE security association negotiation failed because the IKE packet header is invalid.”

What's actually happening

IPsec IKE (Internet Key Exchange) uses UDP port 500 and 4500. The error means the IKE packet arrived at the receiver (the VPN server or your client) but the header structure was corrupt or didn't match what the receiver expected. Two common triggers:

NAT traversal mismatch. Your client is behind a NAT router, but the VPN policy doesn't allow UDP encapsulation. The NAT device rewrites the source port or IP, which corrupts the IKE header checksum.
Registry tweak gone wrong. Some admins manually set AssumeUDPEncapsulationContext for NAT-T, but the value is wrong for the network.

The root cause: Windows IPsec stack rejects the packet because it expects a specific header layout (like a valid SPI or a correct payload length) and the incoming packet doesn't match. The fix usually involves either adjusting the NAT traversal registry key or adding a firewall rule to allow UDP 500/4500 without NAT interference.

Fix it

Check the registry for NAT-T assumptions.
Open Regedit as admin. Navigate to:
```
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent
```
Look for a DWORD called AssumeUDPEncapsulationContext. If it doesn't exist, create it. Set its value to 2 (decimal).
Why 2? 0 means no NAT assumption (default). 1 means the client assumes NAT on the VPN server side. 2 assumes NAT on both sides. The majority of home/office routers do NAT on the client side, and many VPN servers sit behind NAT too — value 2 covers both cases. Reboot after setting this.
Verify UDP 500/4500 are allowed through the Windows Firewall.
Run as admin:
```
netsh advfirewall firewall add rule name="IPsec IKE" protocol=udp localport=500,4500 action=allow dir=in
```
This opens inbound UDP ports 500 and 4500. The real fix here is that some third-party VPN clients (like Cisco AnyConnect) or aggressive Group Policies can block these ports. If you're on a domain, check with your admin — don't override GPO blindly.
Disable IPsec offload on the NIC.
Open Device Manager, right-click your network adapter, choose Properties. Go to the Advanced tab. Look for IPsec Offload, Large Send Offload, or TCP Checksum Offload. Set all of those to Disabled.
The reason: some cheap NICs corrupt IKE headers when offloading encryption. This step alone fixes the error on Realtek and older Broadcom chips.
Flush and restart the IPsec service.
Open Command Prompt as admin:
```
net stop ipsec
net stop ikeext
net start ikeext
net start ipsec
```
This clears any stuck IKE state from previous failed attempts.

If it still fails

Three possibilities:

Your VPN server is asking for a certificate that your client doesn't have. Check the server logs for an IKE authentication payload mismatch. You'll need to install the correct CA certificate.
Your router is doing SIP ALG or VPN passthrough badly. Log into your router and disable any “SIP ALG” or “VPN Passthrough” feature. Those try to inspect and modify UDP packets and often break IKE headers.
The VPN server is running a different IKE version. Windows 10 defaults to IKEv2. If the server only supports IKEv1, you'll need to adjust the connection settings. In the VPN connection properties, under Security, set Type of VPN to “IKEv2” or “L2TP/IPsec” depending on the server config.

One more thing: if you're behind a carrier-grade NAT (like Starlink or 4G/5G hotspots), NAT-T must be enabled on the server. Without it, the IKE header gets mangled by the CGNAT and you'll keep seeing 0x36E0.