Fix 0XC00A0038 Remote Desktop Security Error

This error pops up when Remote Desktop fails to establish a secure connection. It's usually a sign that the encryption or certificate settings are off. Here's how to fix it.

You're sitting at your desk, trying to remote into a machine across the office or a server at a client site. You type in the credentials, hit connect, and instead of a desktop, you get a small dialog: "An authentication error has occurred. The function requested is not supported. STATUS_CTX_SECURITY_LAYER_ERROR (0XC00A0038)". It usually happens right after you enter your username and password, before the connection even attempts to negotiate.

The root cause is simple: the Remote Desktop client and the server can't agree on a security layer. Windows uses TLS (Transport Layer Security) to encrypt the connection, but sometimes the server is set to use a weaker, older encryption protocol like RDP Security Layer (which is basically the old RC4 cipher). When the client is set to require TLS and the server only offers RDP Security, they hit a dead end and throw this error. It's especially common after a Windows update patches the client to be stricter about encryption, or if someone messed with the server's security settings in the registry.

Here's the thing: the fix is rarely on the client side. Even though the error appears on your screen, the problem is almost always on the remote machine. So you need access to that machine's console—either physically or via a different remote tool like VNC or a web-based console—to change the settings. If you're locked out entirely, you might need to boot into Safe Mode or use the recovery environment. Don't panic, it's doable.

Fix 1: Adjust the Server's Security Layer (Registry Method)

This is the most direct fix. On the remote machine, you'll edit the registry to force the security layer to use TLS and set the encryption level to high. Here's how:

Log in to the remote machine physically, or via a console that doesn't use RDP.
Press Win + R, type regedit, and hit Enter.
If User Account Control prompts you, click Yes.

Navigate to this key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp

In the right pane, look for a value named SecurityLayer. If it's not there, right-click, select New → DWORD (32-bit) Value, and name it SecurityLayer.
Double-click SecurityLayer and set the value to 2. That means TLS 1.0 or later. A value of 1 is the older RDP Security Layer, which is likely your problem. Set it to 2.
Now find MinEncryptionLevel. Same routine: if it's not there, create a DWORD named MinEncryptionLevel. Set it to 3, which is High encryption (128-bit).
Click OK and close the Registry Editor.
Reboot the machine. That's non-negotiable—the Terminal Services service needs to restart to pick up the new settings.

After the reboot, try connecting again from your client. If the error persists, move to the next fix.

Fix 2: Use Group Policy to Force TLS

If your remote machine is part of a domain, someone might have set a policy that's overriding the local settings. Here's how to check:

On the remote machine, press Win + R, type gpedit.msc, and hit Enter. (This works on Professional, Enterprise, and Server editions; if you have Home, skip to Fix 3.)
Go to Computer Configuration → Administrative Templates → Windows Components → Remote Desktop Services → Remote Desktop Session Host → Security.
Double-click "Require use of specific security layer for remote (RDP) connections".
Set it to Enabled, and under Options, change the Security Layer to SSL (TLS 1.0).
Click OK.
Also check "Set client connection encryption level" in the same folder. Set it to Enabled and pick High Level.
Close Group Policy Editor and run gpupdate /force in an elevated command prompt. Then restart the machine.

If you're not on a domain, Group Policy might still be affected by local policies, but the registry fix above usually overrides that. Still, it's worth checking both.

Fix 3: Update or Reset the RDP Client Certificate

Sometimes the error isn't about the security layer but about the certificate itself. The Remote Desktop server uses a self-signed certificate for TLS, and if that certificate is corrupted or expired, you'll get exactly this error. Here's how to force a new one:

On the remote machine, open an elevated Command Prompt (right-click → Run as administrator).
Run this command to delete the existing certificate:
```
certutil -delstore "Remote Desktop"
```
Don't worry, it won't ask for confirmation. It just deletes the certificates in the Remote Desktop store.
Next, run:
```
net stop termservice
```
This stops the Terminal Services service.
Restart it with:
```
net start termservice
```
The service will generate a new self-signed certificate automatically when it starts.
Test the connection again.

This fix is a bit more aggressive, but it's saved my butt more than once on Windows Server 2016 and 2019 machines where the cert had been expired for months.

Fix 4: Check TLS Settings on the Server (Advanced)

If the above doesn't work, the server might have TLS 1.0 disabled, and the client is trying to negotiate a version that the server doesn't support. Yes, it's counterintuitive, but the error message is vague. Here's what to check:

On the remote machine, open the Registry Editor again.

Navigate to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols

If you see a key for TLS 1.0, expand it, and check Client and Server subkeys. If DisabledByDefault is set to 1 in the Server subkey, TLS 1.0 is off. Common on hardened machines.
To re-enable, delete the Enabled value (if it's set to 0) and set DisabledByDefault to 0. Or simply delete the entire TLS 1.0 key if you're comfortable—the OS will use system defaults.
Reboot again. Yes, again. I know it's annoying, but TLS changes require a restart.

In my experience, this is rarely needed unless someone has been messing with SCHANNEL settings for security compliance. But if you're stuck, it's the next thing to check.

What If It Still Fails?

Here's the checklist I run when I've done all of the above and I'm still staring at 0XC00A0038:

Check the firewall on the remote machine. Port 3389 needs to be open, but also make sure there's no third-party firewall (like McAfee or Norton) blocking the connection at a protocol level.
Check the client's own TLS settings. On your Windows 10/11 machine, make sure TLS 1.0 is enabled in Internet Options → Advanced. Sometimes a security suite disables it.
Try a different client. Use the Microsoft Store version of Remote Desktop (if you're on a newer Windows) or a third-party client like Microsoft Remote Desktop for Mac. If the error doesn't appear there, the issue is specifically with the built-in client's settings.
Look at the server's event logs. On the remote machine, open Event Viewer → Windows Logs → System and look for TerminalServices or Schannel events around the timestamp of your failed attempt. The details often point to a specific cause like "The certificate chain was issued by an authority that is not trusted."

One last thing—if you're trying to connect to a machine that's not configured for RDP at all (e.g., it's a Home edition or RDP is disabled), you'll get a different error, so make sure RDP is enabled in System Properties → Remote.

That's the whole playbook. The next time you see 0XC00A0038, you'll know exactly what to do. And if it turns out to be the registry fix, you'll have it sorted in under five minutes.