Windows Server 2016 NLA Error: RDP Won't Connect Over VPN

You're sitting at a client site, VPN connected to the office network, and you double-click the RDP file for your Windows Server 2016 box. The connection window flashes, then disappears. Or maybe it hangs at 'Configuring Remote Session' for 30 seconds before dropping with 'An authentication error has occurred. No User Interface.' This is a classic NLA (Network Level Authentication) failure triggered by a certificate mismatch or a missing trusted root CA on the server.

I've seen this exact mess on Server 2016 more times than I can count. The root cause is straightforward: NLA relies on the server presenting a valid SSL certificate to the client during the TLS handshake. When you're connecting over a VPN, the server's certificate might be self-signed, expired, or its CA isn't trusted on the client. Windows 10/11 clients are particularly picky about this. The real fix isn't always the same – it depends on whether you need NLA for security policies or not.

Step-by-Step Fix

Option 1: Disable NLA (Fastest, Works Every Time)

If you don't need NLA for compliance, this is your quickest route. It leaves the server open to credential attacks, so only do this on internal networks or behind a VPN.

1. Log into the server via iDRAC, iLO, or a local console session.
2. Open Server Manager > Remote Desktop Services > Collections (or directly via 'System Properties' if it's a standalone server).
3. Right-click your collection (or go to System Properties > Remote tab) and select Properties.
4. Under Security Layer, change from 'Negotiate' or 'SSL (TLS 1.0)' to RDP Security Layer.
5. Uncheck Require use of Network Level Authentication.
6. Click OK, then restart the Remote Desktop Services service from Services.msc or run net stop termservice && net start termservice.
7. Try connecting again via VPN. It should work now.

Option 2: Fix the Certificate Trust (Keep NLA)

For environments where NLA is non-negotiable, you need to ensure the server's RDP certificate is trusted by the client. Server 2016 often auto-generates a self-signed certificate that the client doesn't trust.

1. On the server, open Remote Desktop Services Manager > right-click your server name > Properties > Certificates tab.
2. Look at the certificate used for RDP over TLS. If it's self-signed (Issued to equals Issued by), you need to replace it. You can use a CA-issued cert from your internal PKI, or import the self-signed cert's root to the client.
3. Option A – Use a CA cert: Deploy a certificate from your internal CA with the server's FQDN as the CN. Assign it in the same Certificates tab.
4. Option B – Trust the self-signed cert: Export the server's self-signed cert from Certlm.msc > Remote Desktop folder > right-click > All Tasks > Export. Copy the .cer file to the client machine and import it into Trusted Root Certification Authorities store for the local machine.
5. On the client, run certlm.msc (needs admin rights), go to Trusted Root Certification Authorities > Certificates, right-click > All Tasks > Import and select the .cer file.
6. Restart the Remote Desktop Services on the server and try the connection again.

Option 3: Update CredSSP (Windows 10/11 Client Side)

Sometimes the issue is the CredSSP update from Microsoft's 2018 patch. If the server hasn't been updated, you can force the client to use an older version.

1. On the Windows client, open Group Policy Editor (gpedit.msc) or local policy.
2. Go to Computer Configuration > Administrative Templates > System > Credentials Delegation.
3. Enable Encryption Oracle Remediation and set it to Vulnerable.
4. Apply and reboot the client.
5. Try the RDP connection again.

What to Check If It Still Fails

If none of that works, you're likely dealing with a weird edge case. Check these:

• VPN MTU issues. Run a ping with ping -f -l 1472 server_ip from the client. If it fails, lower the MTU on the VPN adapter to 1400.
• DNS resolution. Make sure the client resolves the server's FQDN to the correct IP over the VPN. Use nslookup server_fqdn.
• Firewall rules. Port 3389 (TCP) must be open both ways. Check Windows Firewall and any third-party firewall.
• RDP listener bindings. Open an admin command prompt and run netstat -ano | findstr :3389. If the listener isn't on 0.0.0.0 (e.g., it's bound to a specific IP that's not the VPN IP), that's your problem. You can fix it via the registry at HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp and set fUseDefaultNetworkInterface to 1.

I've burned hours on this exact error. Nine times out of ten, disabling NLA or fixing the certificate trust resolves it. Don't overthink it. Start with Option 1 if you can afford the security trade-off. If you can't, chase the certificate chain until it's clean.