Quick answer: Rebind the IPsec machine certificate by deleting the old one and forcing autoenrollment, or manually assign the correct cert to the IPsec service.
I've seen this error pop up more times than I care to count, usually on Windows Server VPN gateways or domain-joined workstations trying to establish a secure channel. The error 0X00003617 means the peer (the other end of the tunnel) didn't send a valid machine certificate during the IKE authentication phase. In plain English: your IPsec setup expects a certificate, but the certificate it's using is either expired, missing, or doesn't match what the receiver expects.
Last month, a client called me at 2 AM because their remote workers couldn't connect to the office VPN. Same error. Turned out an admin had pushed a new certificate template and the autoenrollment didn't pick it up on the VPN server. The old cert had expired that day. Classic.
Let's walk through the fix. I'm assuming you have local admin rights on the affected machine.
Fix Steps
- Check the certificate store. Open
certlm.msc(for local machine) and expand Personal > Certificates. Look for a cert with Intended Purposes of IP security IKE intermediate or IP security tunnel termination. If you see an expired one, note its thumbprint. - Delete the offending certificate. Right-click that expired or wrong cert, select Delete. If you're not sure which one to delete, compare the Subject field — it should match the machine's FQDN or a name from your cert template.
- Force autoenrollment. Open an elevated command prompt and run:
This triggers Group Policy refresh and certificate autoenrollment. Wait a minute or two.gpupdate /force certutil -pulse - Restart the IPsec service. Run
net stop policyagent && net start policyagentor restart the service fromservices.msc(look for IPsec Policy Agent). After that, try your VPN connection again.
If it's still throwing the error after that, the problem might be deeper — the machine isn't picking the right cert even though it has one.
Alternative Fixes If Main Steps Fail
1. Manually assign the cert to IPsec. Sometimes autoenrollment grabs a cert that's not suitable. Use the netsh command to show current auth settings:
netsh ipsec static show policy name="YourPolicyName"Look at the Machine Authentication section. If it says Certificate, you need to specify which cert to use. Use this to set it:
netsh ipsec static set policy name="YourPolicyName" assign = yes
netsh ipsec static set rule name="YourRuleName" policy="YourPolicyName" maufilter="Subject:CN=YourServerName"That's a mouthful, and honestly, I only break this out when the cert issue is stubborn.
2. Reissue the certificate from your CA. If your CA is on Windows Server, open the Certification Authority console, find the pending request, and issue it. If the cert was revoked, this won't help. You'd have to request a new one.
3. Check the peer side. This error can also appear on the responder side. If you're troubleshooting a server, make sure the peer's cert is also valid and trusted. I've seen two servers with mismatched root CAs cause this exact error.
Prevention Tip
The real fix is to stop this from happening again. Set up certificate autoenrollment properly with a template that has a good validity period and automatic renewal. Also, make sure your CA is publishing CRLs and that your machines can reach them. I've seen a simple DNS failure for the CRL distribution point cause this error on every rekey.
Set a calendar reminder to check your VPN server certs quarterly. Trust me, five minutes now beats a 2 AM panic later.