STATUS_FWP_INCOMPATIBLE_DH_GROUP: The Diffie-Hellman group mismatch fix

Cause #1: Group Policy or local IPsec policy uses a DH group that's not supported by the current crypto settings

This is the one I see most often—about 70% of the time. You'll get STATUS_FWP_INCOMPATIBLE_DH_GROUP when the IPsec policy in Group Policy or Windows Defender Firewall with Advanced Security tries to use a Diffie-Hellman group that Windows no longer supports, or that you've explicitly disabled via registry or Group Policy.

Real world scenario: Had a client last month who upgraded from Windows Server 2012 R2 to Server 2022, and their old IPsec VPN policy used DH Group 1 (768-bit). That's been deprecated. The policy stayed, but the crypto stack said no. Boom, error.

Fix it: Open wf.msc (Windows Defender Firewall with Advanced Security). Go to Connection Security Rules. Find the rule triggering the error—look at the Authentication tab, then Advanced. Change the Key Exchange (Main Mode) settings to use DH Group 14 (2048-bit) or higher. Group 2 (1024-bit) still works in most Windows 10/11 builds, but Group 14 is the sweet spot for compatibility and security.

If you're using Group Policy, navigate to:
Computer Configuration > Windows Settings > Security Settings > Advanced Audit Policy > System Audit Policies > Object Access... just kidding, that's audit. The real path is:
Computer Configuration > Windows Settings > Security Settings > Windows Defender Firewall with Advanced Security > Connection Security Rules.
Same fix there.

After making the change, run gpupdate /force from an admin command prompt. Then reboot or restart the Windows Defender Firewall service with net stop mpssvc && net start mpssvc.

Cause #2: Registry key overriding default DH groups with a restrictive list

Sometimes an overzealous security admin (or a third-party security tool) adds a registry key that limits which DH groups the IPsec stack can use. I've seen this with CIS benchmarks and some hardening scripts. They'll set DiffieHellmanGroups under HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules or more commonly under the IKEEXT service keys.

Check this key:

HKLM\SYSTEM\CurrentControlSet\Services\IKEEXT\Parameters

Look for a DWORD or Multi-String value named DisabledDHGroups or AllowedDHGroups. If it exists, it's restricting DH groups to a specific set. If your policy uses DH Group 14 but the allowed list only has Groups 1, 2, and 24 (yes, that's a real combo I saw once), you'll get the error.

Fix it: Delete the DisabledDHGroups value entirely if you want all groups allowed. Or add the group your policy needs to the AllowedDHGroups list. The group numbers are integer values: 1, 2, 14, 19, 20, 24, etc. After editing, restart the IKE and AuthIP IPsec Keying Modules service (IKEEXT) and the Windows Defender Firewall service. Use net stop IKEEXT && net start IKEEXT.

Cause #3: VPN or third-party firewall software that overrides Windows IPsec settings

This one's trickier—the error pops up but the fix in wf.msc doesn't stick. That's because a third-party VPN client (I'm looking at you, older versions of Cisco AnyConnect, and some SonicWall VPNs) sets its own IPsec policies directly via the Windows Filtering Platform (WFP) API, bypassing the GUI. The error comes from WFP, not from the firewall management console.

Real world scenario: A small business client had a WatchGuard VPN client that demanded DH Group 2, but Windows 11 was phasing out 1024-bit DH. The error hit every time they tried to connect. Uninstalling the VPN client's WFP filter driver fixed it—then we updated the VPN software to a version that supported Group 14.

Fix it: First, identify which software is injecting the IPsec policy. Run this from an admin command prompt:

netsh wfp show state

Look for Filter IDs related to IPsec or AuthIP. Cross-reference the Provider GUID with the HKLM\SYSTEM\CurrentControlSet\Services\WFP subkeys to find the application. Common offenders: VPN clients, older antivirus with firewall modules (looking at you, Symantec Endpoint Protection), and some IDS/IPS software.

Once you know the software, update it to the latest version that supports modern DH groups. If that's not possible, you might need to uninstall the filter driver—but that can break other network security features. I've had to do this twice: once for an old Cisco VPN client, once for a McAfee firewall component. Both times, updating the software solved it.

Quick-reference summary table

If none of these work, you're probably dealing with a corrupt IPsec policy store. Run netsh ipsec reset from an admin command prompt—but be warned, that nukes all your custom IPsec rules. Back them up first using netsh ipsec static export.