Fix ERROR_IPSEC_IKE_ENCRYPT (0x0000362A) – Payload Encryption Failure

Quick answer

Flush the IPsec security associations and reset the Windows Security Association database with netsh ipsec static flush run as admin, then restart the VPN service. If the error persists, check that both sides agree on an encryption algorithm (AES-256, not 3DES) and that the preshared key matches exactly.

What's actually happening here

This error shows up when the IKE (Internet Key Exchange) protocol tries to encrypt a payload during phase 1 or phase 2 of IPsec negotiation, but the encryption method fails. Windows throws ERROR_IPSEC_IKE_ENCRYPT (0x0000362A) — literally "Error encrypting payload."

The root cause is almost always a mismatch in the encryption algorithm suite between the two peers. One side wants AES-256 with SHA-256, the other offers only 3DES with SHA-1. They can't agree, so the encryption call fails. I've seen this on Windows 10 build 22H2, Windows 11 23H2, and Server 2022 when connecting to older Cisco ASA firewalls or third-party VPN gateways that have been reconfigured without updating the Windows IPsec policy.

A less common trigger: a corrupted security association (SA) database from a previous failed handshake. The stale SA still references old crypto parameters, and Windows tries to reuse them instead of renegotiating fresh.

1. Flush the IPsec SA database

   Open an elevated Command Prompt (Win+X, Terminal (Admin)) and run:

   netsh ipsec static flush

   This wipes all stored security associations. What happens next: when you retry the VPN connection, Windows must build a new SA from scratch, which forces both sides to renegotiate encryption algorithms. The reason step 3 works is that you remove any stale crypto state that might be causing the encryption call to reuse a bad context.

2. Verify the VPN connection's encryption requirements

   Open ncpa.cpl (Network Connections), right-click your VPN adapter, go to Properties → Security tab. Check "Advanced (custom settings)" and click Settings. Under Encryption, make sure it's set to "Require encryption" — not "Optional" or "No encryption allowed." This forces Windows to negotiate an encrypted session, which is what the remote peer expects.

3. Restart the IPsec services

   net stop PolicyAgent
   net stop IKEEXT
   net start IKEEXT
   net start PolicyAgent

   The PolicyAgent service manages IPsec policy, and IKEEXT handles IKE negotiations. Restarting them in this order reloads the crypto providers from scratch. If a third-party security software (like a firewall or antivirus) hooked into the IKE stack, a restart often clears that interference too.

4. Check the preshared key and authentication method

   Go to the VPN adapter Properties → Security tab → Type of VPN. If you're using L2TP/IPsec with a preshared key, ensure it matches exactly — case-sensitive, no trailing spaces. Then click Advanced settings under Authentication, and confirm that EAP or MS-CHAP v2 is selected appropriately for your server. Mismatched authentication settings can cause the encryption step to fail because the key derivation function receives bad input.

5. Reset the IPsec policy to default

   If the above steps don't fix it, someone may have applied a custom IPsec policy via Group Policy or registry. Run:

   netsh ipsec static show policy

   If any policies appear beyond the default "Request IPsec" or "Require IPsec" entries, remove them with:

   netsh ipsec static delete policy name="PolicyName"

   Then flush again. A stray policy can explicitly block certain encryption algorithms, causing the encryption call to fail when the remote peer doesn't match.

Alternative fixes if the main steps fail

• Disable IPv6 temporarily. I've seen cases where the IKE negotiation on the IPv6 stack gets confused and throws this error. Uncheck IPv6 in the VPN adapter Properties → Networking tab.
• Downgrade to AES-128. Some older VPN gateways (especially on Cisco IOS 15.x) don't support AES-256 in phase 1. Force Windows to use AES-128 by editing the registry at HKLM\SYSTEM\CurrentControlSet\Services\PolicyAgent, create a DWORD AssumeUDPEncapsulationContextOnSendRule set to 2, then restart.
• Install the KB5001404 update (or a later cumulative update) if you're on Windows 10 2004/20H2 — older versions had a known bug where the IKE encryption provider would fail after a sleep/resume cycle.

Prevention tip

Before configuring a new VPN connection, run netsh ipsec static show all to snapshot your existing IPsec policies. That way, if something breaks, you can restore them quickly. Also, make sure your VPN gateway and Windows client both support the same encryption algorithms — I always standardize on AES-256 with SHA-256 for phase 1 and phase 2. It avoids negotiation failures entirely.