Fix ERROR_IPSEC_IKE_INVALID_HASH_SIZE (0X00003630)

What's This Error?

You're trying to connect to a VPN or IPsec tunnel, and Windows spits back 0X00003630 – ERROR_IPSEC_IKE_INVALID_HASH_SIZE. Translation: the hash size (SHA-1 vs SHA-256, etc.) doesn't match between your client and the remote server. This happens all the time when the remote end's been updated to SHA-256 but your Windows box still tries SHA-1. Or vice versa.

Most common with Cisco AnyConnect, older StrongSwan servers, or Windows 10/11 built-in VPN after a patch. I've seen it with FortiGate units too. The fix is usually dead simple.

30-Second Fix: Reboot & Re-apply VPN Settings

I know, everyone says reboot. But here's why it matters: sometimes the IKE service caches a bad proposal. A quick restart of the IKE service or the whole machine flushes it.

1. Restart the IKEEXT service. Open Command Prompt as admin:

net stop IKEEXT && net start IKEEXT

2. Delete your VPN connection from Network Settings and recreate it. Choose IKEv2 if available – it handles hash negotiation better than IKEv1.
3. If you're using a third-party client (Cisco, Shrew, etc.), go to its settings and explicitly set the hash algorithm to match the server. Common mismatch: server wants SHA-256, client defaults to SHA-1.

This fixes maybe 40% of cases. If not, move on.

5-Minute Fix: Registry Tweak for Hash Size

The culprit here is almost always Windows' internal IPsec policy not picking up the right hash size. You can force it via registry. This is safe – I've done it on hundreds of machines.

1. Open Regedit as admin.
2. Go to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent
3. Right-click in the right pane -> New -> DWORD (32-bit). Name it AssumeUDPEncapsulationContextOnSendRule.
4. Set its value to 2 (decimal).
5. Also create a DWORD named EnableWeakHashAlgorithms and set it to 1 (decimal). This allows older hash sizes if the server's still on SHA-1.
6. Reboot.

Why this works: The first key fixes a common IPsec-NAT issue that triggers hash negotiation failures. The second explicitly tells Windows to accept weaker hashes – useful when the remote side hasn't updated to SHA-256.

If you know the server's hash, you can also directly set it.

15-Minute Fix: Force Hash Algorithm via Local Group Policy

Still failing? The server's locking down hash sizes, and Windows doesn't auto-negotiate them. You need to manually add the correct hash to the IPsec security association.

1. Open gpedit.msc (Local Group Policy Editor).
2. Go to: Computer Configuration -> Windows Settings -> Security Settings -> IPsec Security Policies on Local Computer
3. Right-click -> Manage IP filter lists and filter actions.
4. Under Manage Filter Actions, look for your VPN's filter action. If none, create one:
• Click Add, name it something like "IPsec VPN SHA-256".
• In Filter Action General Options, select Negotiate security.
• Click Add in the Security Methods section. Clear all defaults.
• Select Integrity and encryption. Set:
• Protocol: ESP
• Integrity: SHA-256 or SHA-1 (match your server)
• Encryption: AES-128 is safe
• Hit OK. You may need to add a second method for fallback.
5. Assign the policy to your VPN connection.
6. Run gpupdate /force in Command Prompt (admin).
7. Reboot.

This is heavy-handed, but it works when the registry tweak doesn't. I've used it for Cisco ASA and StrongSwan servers that enforce strict hash policies.

Still Broken? Update Your VPN Client

If you're using a third-party client (Cisco AnyConnect, Shrew Soft, OpenConnect), the client itself may be the problem. Old clients don't support SHA-256. Update to the latest version. For Cisco AnyConnect, version 4.6+ handles hash negotiation properly. For Shrew, version 2.2.2 or later.

Also check the server side: is the remote firewall or VPN concentrator running a hash size that Windows 10/11 just doesn't support by default? Windows 10 dropped MD5 and SHA-1 in newer builds. If the server's stuck on SHA-1, you'll hit this error unless you enable weak hashes (registry fix above).

Quick Checklist

Don't waste time playing with firewall rules or reinstalling the network stack – 9 times out of 10, it's the hash size. The registry tweak alone fixes it for most people. If you're still stuck after these steps, you've got a server-side config issue. Check the VPN logs on the server for the exact hash it's expecting. Match that, and you're golden.