Fix ERROR_IPSEC_IKE_INVALID_CERT_KEYLEN 0x00003639
Windows throws this when your VPN or IPsec certificate key is too short for the security policy. Usually a 1024-bit RSA cert - you need 2048-bit or higher.
The 30-Second Fix: Bypass the Key Length Check
I know this error is infuriating. You're trying to connect, the certificate looks fine, and then Windows throws 0x00003639 right in your face. The root cause is almost always a certificate with a 1024-bit RSA key when your IPsec policy demands at least 2048 bits. But if you're in a test environment or you know the 1024-bit cert is from a trusted source, there's a quick registry override that tells Windows to chill out.
Open regedit (Win+R, type regedit, hit Enter). Navigate to:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\ParametersRight-click in the right pane, choose New > DWORD (32-bit) Value. Name it AllowCertificateWithWeakKeyLength. Set the value to 1. Reboot your machine or restart the Routing and Remote Access service. Try the VPN or IPsec connection again.
This tripped me up the first time too, because it works instantly. But it's a workaround, not a real fix. Only use it if you trust the source of that certificate and understand the security trade-off. For production environments, skip this step and move on to the real fix.
The 5-Minute Fix: Check and Update Your IPsec Policy
So the registry tweak didn't work, or you don't want to weaken security. The next stop is the IPsec security association (SA) settings. Windows Server 2016, 2019, and 2022 default to requiring 2048-bit keys for IKEv2. That's fine, but sometimes Group Policy or an admin pushed a custom policy that has even stricter requirements - like 3072 or 4096.
Open an elevated PowerShell prompt (right-click Start, choose Windows PowerShell (Admin)). Run:
Get-VpnServerConfiguration | Select-Object *Look for properties like IKEv2CertificateKeyLength or CertificateKeyLength. If it's set to anything above 2048, and your certificate is 2048-bit, that's your mismatch. Change it with:
Set-VpnServerConfiguration -IKEv2CertificateKeyLength 2048For Windows 10/11 clients connecting to a VPN, check the client-side policy. Open the local Group Policy editor (gpedit.msc). Go to Computer Configuration > Administrative Templates > Network > Network Connections > Windows Filtering Platform. Look for a setting that enforces minimum key lengths. Disable it or set it to match your cert.
If you're connecting to a third-party VPN like a Cisco ASA or Palo Alto, the error can also pop up because the remote side demands a longer key. Check the VPN server's IPsec proposal settings. On a Cisco ASA, that's crypto ikev2 policy. Make sure your certificate key length matches the proposal.
The 15+ Minute Fix: Replace the Certificate with a Proper Key
This is the real fix. If your certificate has a 1024-bit key, get one with at least 2048 bits. Here's how to generate a proper self-signed certificate for internal use on Windows Server.
Open PowerShell as Administrator. Run this command to create a new self-signed certificate with a 2048-bit key, valid for 5 years:
New-SelfSignedCertificate -Type Custom -Subject "CN=YourVPNCert" -KeyUsage DigitalSignature -KeyAlgorithm RSA -KeyLength 2048 -CertStoreLocation "Cert:\LocalMachine\My" -NotAfter (Get-Date).AddYears(5)Note the thumbprint from the output. Then export it with the private key:
$password = ConvertTo-SecureString -String "YourPassword123" -Force -AsPlainText
Export-PfxCertificate -Cert "Cert:\LocalMachine\My\[Thumbprint]" -FilePath "C:\temp\vpn.pfx" -Password $passwordImport that PFX into the Local Machine > Personal store on the client machine. Then update your VPN connection to use this new cert. On Windows 10/11, go to Settings > Network & Internet > VPN > [Your VPN] > Advanced options > Edit > Certificate and pick the new one.
If you're using a CA-issued certificate (internal or public), you need to request a new one with a 2048-bit key. For an internal Active Directory Certificate Services CA, submit a certificate request with key length 2048. For a public CA like DigiCert or Let's Encrypt, check your certificate order - most issue 2048-bit by default now, but older certs or specific profiles might still be 1024.
One last thing: verify the old cert is actually the problem. Open certlm.msc (Local Machine certificates), find the VPN certificate, double-click, go to the Details tab, and look at Public key. If it says RSA (1024 Bits), that's your culprit. If it says 2048 or higher, then the issue is somewhere else - check the IPsec policy again, or look at the remote server's setting.
I've seen this error in two specific scenarios: a Windows 10 laptop trying to connect to a Server 2019 RRAS VPN with a leftover 1024-bit cert from a domain migration, and a third-party VPN client (Shrew Soft) using a certificate from a public CA that had a 1024-bit intermediate. The fix was the same in both cases: get a 2048-bit cert into the machine store.
If you're still stuck after all this, look at the event logs. Open Event Viewer, go to Applications and Services Logs > Microsoft > Windows > RasRouting > Operational. Look for event ID 20057 - that usually gives you the exact certificate hash Windows is rejecting. Cross-reference that with the certificates in certlm.msc to find the bad one.
Was this solution helpful?