Fix ERROR_IPSEC_IKE_PROCESS_ERR_NONCE (0X0000360F)

When This Error Hits

You're setting up a site-to-site or client VPN using IPsec with IKEv1 or IKEv2. The connection starts — you see Phase 1 begin — then it dies with ERROR_IPSEC_IKE_PROCESS_ERR_NONCE (0x0000360F). The event log shows it right after the IKE initiator sends its first nonce. I've seen it most often when a router or firewall appliance sits between the two VPN peers and inspects or alters UDP 500 / 4500 traffic.

Root Cause

The nonce is a random number each side sends during IKE negotiation. It's used to prevent replay attacks and to derive session keys. If the responder can't parse the nonce payload — either because it's truncated, corrupted, or the packet got fragmented and reassembled wrong — it throws this error. The culprit here is almost always one of three things:

• NAT-T (NAT Traversal) is misconfigured or missing on one side.
• An intermediate device (firewall, router, IPS) is inspecting or modifying UDP 500/4500 packets.
• MTU issues cause fragmentation, and the nonce payload arrives in pieces.

Less common but worth mentioning: mismatched IKE versions (v1 vs v2) or incorrect Diffie-Hellman groups can also produce this error, but those usually trigger different error codes. If you're getting 0x0000360F specifically, start with the network path.

Step-by-Step Fix

1. Check NAT-T Settings

Both VPN peers must agree on NAT-T. On Windows, it's enabled by default for IKEv2, but for IKEv1 you might need to force it. Run this on the Windows machine:

netsh advfirewall consec show rule name="YourRuleName" verbose

Look for AuthEndpoint. If it's not set, add NAT-T through the firewall rule or registry. For a quick test, enable NAT-T globally on both peers if possible.

2. Verify UDP 500 and 4500 Are Open End-to-End

Don't bother checking just the Windows firewall. Use a tool like Test-NetConnection or telnet from each side to confirm ports are reachable. Even better, run a packet capture on both ends. Look for IKE packets arriving intact. If you see fragmented UDP datagrams, that's your sign.

3. Adjust MTU or Enable IPsec Fragmentation

Routers sometimes drop fragments of UDP packets. On Windows, you can set the IPsec MTU manually:

1. Open regedit.
2. Go to HKLM\SYSTEM\CurrentControlSet\Services\PolicyAgent.
3. Create a DWORD EnableFragmentationCheck and set it to 0.
4. Reboot or restart the IPsec service.

This tells Windows to not fragment IKE packets. If the path MTU is too small, you'll need to reduce it on the router or use TCP instead of UDP (not ideal).

4. Disable Inspection on Firewalls

If you control the firewall between the peers, turn off any deep packet inspection for UDP 500/4500. I've seen Cisco ASA, Fortinet, and even simple consumer routers corrupt the nonce payload during inspection. Create an exception for all traffic to/from both VPN endpoints on those ports.

5. Match IKE Parameters Perfectly

Even though nonce errors are usually network-related, double-check your encryption and hash algorithms match on both sides. Use the same Diffie-Hellman group (Group 2 or 14 are safest). If one side uses SHA256 and the other SHA1, you might get a nonce parsing error because the payload length changes. Here's a common pair that works:

• Encryption: AES256
• Hash: SHA256
• DH Group: 14
• IKE version: IKEv2 (avoid IKEv1 if possible)

What to Check If It Still Fails

Sometimes the error is a red herring. I've spent hours chasing nonce issues only to find a certificate expired or a preshared key typo. If you've done the steps above and it still fails:

• Look at the event logs on both peers. The error may report 0x0000360F but the real cause is logged right before.
• Enable IKE logging on Windows using netsh ras set tracing * enabled. Parse the traces — look for lines mentioning "nonce" or "payload length mismatch".
• Test with a different VPN peer temporarily. Swap in a known-good device (like a simple strongSwan VM) to rule out your main endpoint being the problem.
• If all else fails, restart the IPsec service on both sides and re-enter the preshared key. I've seen a stale security association (SA) cause this even when configs look clean.

One last thing: don't assume both sides use the same NAT-T setting. Many firewalls have NAT-T disabled by default. Check it on the remote side too.