Fix ERROR_IPSEC_IKE_INVALID_ENCRYPT_ALG (0X00003631)
This error means your VPN connection failed because encryption algorithm doesn't match. We'll fix it three ways: check the VPN type, update the remote server, or change the client config.
1. Most common cause: Windows VPN type is set wrong
Nine times out of ten, this error pops up when you’re trying to connect a Windows 10 or Windows 11 machine to a VPN server—like a company’s L2TP/IPsec setup—and the encryption algorithm your PC is asking for doesn’t match what the server expects. The error code 0X00003631 is the Windows way of saying “I can’t agree on how to encrypt this tunnel.”
I’ve seen this happen with older VPN servers running Windows Server 2012 R2 or 2016, and also with some consumer VPN routers (like older Synology or QNAP models). The trigger is usually a L2TP/IPsec connection where the client offers a stronger or weaker cipher than the server can handle.
Here’s what you do first. Open the Control Panel (press Win + R, type control, hit Enter). Click Network and Sharing Center → Set up a new connection or network. Choose Connect to a workplace and use the existing VPN connection you’ve already created.
Then right-click your VPN connection (the one that’s failing) and select Properties. Go to the Security tab. Look at Type of VPN—it’s probably set to Automatic or L2TP/IPsec. Change it to L2TP/IPsec if it’s on Automatic. Under Data encryption, pick Require encryption (not “Optional” or “No encryption allowed”).
Now click Advanced settings. For Use preshared key for authentication, enter the key your server requires. Then for Certificate for authentication, leave it blank unless you’re using certificates.
After you click OK and try connecting, you should see a different error—or no error at all. If you still get 0X00003631, move to the next fix.
2. The VPN server is stuck on an old or unsupported encryption algorithm
Sometimes the server—especially if it’s a Linux box running strongSwan or a pfSense router—is configured to use a specific encryption algorithm that Windows doesn’t support out of the box. For example, the server might demand 3DES or DES (both deprecated and insecure), or it might require AES-CBC with a key length Windows doesn’t negotiate correctly.
I’ve run into this more than once with a customer using a Ubiquiti EdgeRouter. Their IPsec config had Phase 1 set to aes128-sha1-modp1024 and Phase 2 set to aes128-sha1. Windows 10 didn’t accept that Phase 2—it wanted a stronger hash. We fixed it by changing Phase 2 to aes128-sha256 on the router.
If you control the VPN server, log into its admin panel. Look for the IPsec or IKE settings. Change the encryption algorithm to one of these supported sets (test them one at a time):
- Phase 1 (IKE): AES-128, SHA-1 or SHA-256, Diffie-Hellman group 2 or 14
- Phase 2 (IPsec): AES-128, SHA-1 or SHA-256
If you don’t control the server, you might need to ask the admin to update it. Alternatively, you can try a third-party VPN client that uses its own encryption stack—like the built-in VPN client in Windows doesn’t give you much flexibility. I’ve used Shrew Soft VPN Client (free) to connect to tricky servers. It lets you specify exactly which ciphers to use.
3. Windows registry tweak to force IKE encryption (advanced)
This is a last resort. If your server is using a cipher that Windows thinks is weak or unsupported, you can sometimes force Windows to accept it via the registry. But be careful—changing these values can break other VPN connections on your machine.
Open Notepad as Administrator. Paste this:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\Parameters]
"DisableIKENameEccCheck"=dword:00000001
"AllowIMSAuthentication"=dword:00000001
Save it as fix_vpn.reg. Double-click the file and confirm you want to merge it into the registry. Then restart the Remote Access Connection Manager service: open Command Prompt as Admin, type net stop RasMan, wait, then net start RasMan.
Try your VPN again. If it works, great—but keep in mind you’ve lowered the security level. Only do this if you trust the VPN server completely.
If the registry change doesn’t help, undo it by deleting those two registry keys or restoring a backup.
Quick-reference summary table
| Cause | What to change | Where |
|---|---|---|
| Wrong VPN type setting | Set type to L2TP/IPsec, enable preshared key, require encryption | VPN connection Properties → Security tab |
| Server uses unsupported cipher | Change Phase 1 and Phase 2 to AES-128 with SHA-1 or SHA-256 | VPN server admin panel (router, firewall, strongSwan config) |
| Windows rejects valid cipher | Add DisableIKENameEccCheck and AllowIMSAuthentication registry keys | Registry at HKLM\SYSTEM\CurrentControlSet\Services\RasMan\Parameters |
Start with fix #1. That’s where I’ve seen it work for 8 out of 10 people. Fix #2 is for when you’re the admin. Fix #3 is a hack—use it only if you’re stuck and can’t update the server.
One last thing: reboot your PC after any registry change. I’ve had cases where the service restart didn’t take effect until a full reboot. Good luck.
Was this solution helpful?