Fix FWP_E_INCOMPATIBLE_AUTH_CONFIG (0x80320038) in Windows

Quick answer: Open Windows Firewall with Advanced Security (wf.msc), go to Connection Security Rules, find the rule that's failing, and change its authentication method to match what the other side expects — typically Kerberos v5 or certificate.

What's happening here

This error shows up when Windows tries to set up an IPsec security association (SA) and the two sides can't agree on how to authenticate. Think of it like two people at a secure door — one expects a fingerprint scan, the other wants a keycard. Neither gives, and the door stays locked.

I've run into this most often on domain-joined machines trying to talk to a VPN server that expects certificate-based auth, but the Windows rule is set to Kerberos (or vice versa). Another common spot: custom firewall rules you've imported or created that inherit an auth method that doesn't exist on the other side. Had a client last month whose entire remote access setup stopped because someone copied an IPsec rule from a test box without changing the auth type.

Step-by-step fix

1. Open Windows Firewall with Advanced Security — hit Win+R, type wf.msc, press Enter.
2. Click Connection Security Rules (left pane). You'll see a list of rules that define IPsec behavior.
3. Find the offending rule — look for any rule with a red X or that's enabled but not working. If you're not sure which one, check the timestamp or the description. VPN-related rules often say "Remote Access" or "DirectAccess" in the name.
4. Right-click the rule and choose Properties.
5. Go to the Authentication tab. This is where the mismatch lives.
6. Change the authentication method to match what the other side expects:
   
   • If both machines are domain-joined and talking to a domain controller: choose Computer and User (Kerberos v5).
   • If you're using a certificate from a trusted CA (like a VPN server cert): choose Advanced, then click Add, select Certificate, and specify the root CA.
   • If it's a simple test or isolated network (not recommended for production): Preshared Key works, but make sure both sides have the exact same key.
7. Apply and OK. Then click Action > Refresh to reload the rules.
8. Test the connection again. If it still fails, proceed to the alternatives.

If the main fix doesn't work

Alternative 1: Check the other side's configuration

It's not always Windows. On the server or remote gateway, open its IPsec settings. If it's a Cisco ASA, for example, run show crypto ipsec sa and look for the auth method. If it says "pre-shared-key" and your Windows rule says "Kerberos", that's your mismatch. Adjust Windows to match, or change the server side.

Alternative 2: Reset IPsec policy to defaults

Sometimes a corrupted policy or leftover GPO causes this. From an admin PowerShell:

netsh ipsec reset
netsh advfirewall reset

This wipes all custom IPsec rules and firewall settings, so back up your rules first with netsh advfirewall export "C:\backup.wfw". After the reset, rebuild only the rules you need.

Alternative 3: Check Group Policy

If you're on a domain, IPsec settings might be pushed via GPO. Run gpresult /h gp.html, open the HTML file, and look under Computer Configuration > Windows Settings > Security Settings > IP Security Policies. If a policy is enabled, it overrides anything you set locally. Work with your domain admin to fix the GPO — or remove the computer from the GPO scope if it's a test machine.

Alternative 4: Disable IPsec for specific traffic (last resort)

If you just need it working and can't sort out the auth mismatch, you can create a rule that does not require authentication for that specific traffic. In the rule's Properties, Authentication tab, select Do not authenticate. This is insecure — anyone can spoof traffic — but it unblocks you while you figure out the real fix.

Prevention tip

Before you create an IPsec rule, know what auth method the other side uses. A 30-second check saves you an hour of troubleshooting. If it's a third-party VPN appliance, the manual usually spells out expected auth. Document your rules — even a simple text file with the rule name, auth type, and date helps. I keep a sticky note on my monitor for the three environments I support. Sounds old-school, but it's saved me more than once.