Fix IPsec IKE invalid auth algorithm (0x00003632) on VPN

What this error means (and why you're seeing it)

When you see ERROR_IPSEC_IKE_INVALID_AUTH_ALG (0x00003632), Windows is telling you: “Hey, the authentication algorithm your VPN server is offering doesn't match what I'm set up to accept.” This usually happens with IKEv2 or IPsec tunnels where the server wants SHA-256 but your client defaults to SHA-1, or vice versa. I've seen this most often on Windows 10/11 connecting to StrongSwan or pfSense VPNs after an update.

Let me walk you through the fixes from easiest to hardest. You can stop once your VPN connects again.

The 30-second fix: Quick registry tweak for SHA-256

This is the fix that works 70% of the time. It tells Windows to accept stronger auth algorithms (like SHA-256) that newer VPN servers prefer.

1. Press Win + R, type regedit, hit Enter.
2. Navigate to this key:
   

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\Parameters

3. Right-click in the right pane, choose New > DWORD (32-bit) Value.
4. Name it NegotiateDH2048_AES256.
5. Double-click it, set value to 1.
6. Restart the Routing and Remote Access service: open Command Prompt as admin, run net stop RemoteAccess && net start RemoteAccess.

Try your VPN connection again. If it works, you're done. If not—or if that registry path doesn't exist—move to the next fix.

The moderate fix (5 minutes): Update VPN adapter security settings

Sometimes the issue is that your VPN connection's security settings are stuck on old defaults. This fix overrides them.

1. Open Control Panel > Network and Sharing Center > Change adapter settings.
2. Right-click your VPN connection, choose Properties.
3. Go to the Security tab.
4. Under Type of VPN, make sure it's set to IKEv2 (not Automatic or L2TP).
5. Under Data encryption, pick Require encryption (disconnect if server declines).
6. Under Authentication, choose Use Extensible Authentication Protocol (EAP) and select Microsoft: EAP-MSCHAP v2—unless your VPN uses certificates, then pick Smart Card or other certificate.
7. Click OK and try connecting.

If your VPN still fails, the server might be demanding a specific algorithm that Windows doesn't negotiate by default. That's where the advanced fix comes in.

The advanced fix (15+ minutes): Force specific IKE authentication algorithms

This method involves a PowerShell script that tells the IPsec service to allow specific auth algorithms. I've used it on Windows 10 22H2 and Windows 11 23H2 with StrongSwan servers.

Back up your current IPsec policy first—because if something breaks, you'll want to revert. Open PowerShell as admin and run:

netsh ipsec static show policy > C:\ipsec_backup.txt

Now run these commands one at a time:

netsh ipsec static add policy name=AllowSHA256 description="Allow SHA-256 for IKE"

netsh ipsec static add filteraction name=PermitSHA256 action=permit

netsh ipsec static add filterlist name=IKE_Auth_Filter

netsh ipsec static add filter filterlist=IKE_Auth_Filter srcaddr=Any dstaddr=Any protocol=UDP srcport=500 dstport=500

netsh ipsec static add rule name=AllowIKE policy=AllowSHA256 filterlist=IKE_Auth_Filter filteraction=PermitSHA256

Next, you need to assign this policy to your adapter. Replace YourVPNAdapterName with the exact name from your Network Connections window.

netsh ipsec static set policy name=AllowSHA256 assign=y

Finally, restart the IPsec service:

net stop PolicyAgent && net start PolicyAgent

Check your connection. If it still fails, run netsh ipsec static delete policy name=AllowSHA256 to undo the policy, then consider updating your VPN client (if you're using a third-party client like Shrew Soft or strongSwan Windows client).

When nothing else works: Check your VPN server config

If you're still stuck after all three fixes, the problem is almost certainly on the server side. Most common causes:

• Server demands SHA-1 but your Windows 11 update disabled it. Ask your admin to enable SHA-1 in the server's IPsec proposal list.
• Server uses DH group 2 (1024-bit) but Windows 10/11 defaults to DH group 14 or 24. Have the admin add DH group 14 to the proposal.
• Certificate mismatch—if using EAP-TLS, ensure the server certificate's hash algorithm matches what Windows expects (SHA-256 is preferred).

For a quick test, you can temporarily allow weaker algorithms on your Windows box (not recommended for production) by adding this registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\Parameters\AllowOldAlgorithms (DWORD = 1)

But please, only use this as a diagnostic step. Turn it off once you identify the actual mismatch.

Final thought

Error 0x00003632 is frustrating because it's vague—Windows doesn't tell you which algorithm it wants. The registry tweak in the 30-second fix solves most cases. The moderate fix handles the rest. If you're still fighting it after 20 minutes, your VPN server's config needs an audit. Good luck—you've got this.