SEC_E_INCOMPLETE_MESSAGE (0x80090318) — Fix for TLS Handshake Failures

You're working, and suddenly Outlook says it can't connect to the server. Or maybe your SQL Server Management Studio throws a timeout when trying to link to a remote database. Crack open the Event Viewer, and there it is: Schannel event ID 36888 with SEC_E_INCOMPLETE_MESSAGE (0x80090318) — "The supplied message is incomplete."

I've seen this exact thing on a client's Windows Server 2019 last month. Their ERP system used a custom LDAP-over-TLS connection to a domain controller, and it kept dropping every 30 minutes. Same error. The fix wasn't obvious, but it's repeatable.

Here's what's happening: When Windows tries to establish a secure TLS connection, it expects the server to send a complete TLS handshake record in one go. If the server sends the data in smaller chunks (like over a slow network, or behind a misconfigured proxy or load balancer), Schannel bails out with this incomplete message error. It's a client-side timeout, not a server problem. The server thinks it's sending data fine, but Windows gives up waiting for the full packet.

Root Cause

The core issue is a Schannel timeout defaulting to 10 seconds. If the server doesn't complete the TLS handshake in that window — maybe it's fragmenting the certificate chain, or the network has latency — Windows drops the connection. This is especially common with:

• Outlook 2016/2019 connecting to Exchange 2016 with TLS 1.2
• SQL Server linked servers using SSL encryption
• Custom apps using .NET's SslStream with large certificates
• LDAP over TLS to domain controllers behind a VPN

Fix It: Increase the Schannel Timeout

1. Open Regedit as Administrator.
2. Navigate to:

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel

3. Create a new DWORD (32-bit) value named MessageTimeout. Right-click the Schannel folder → New → DWORD (32-bit) Value.
4. Set the data: Double-click it, set to 30000 (that's 30 seconds). Decimal. This gives the server 30 seconds to complete the handshake. Don't go crazy — 60 seconds max, or you risk hanging your app.
5. Reboot the machine. This isn't a live-updating key — Schannel reads it at boot.

Other Possible Fixes (If Still Failing)

If the timeout increase didn't do it, the problem is something else. Here's the checklist:

1. Check TLS Version Mismatch

Windows 7/Server 2008 R2 may not have TLS 1.2 enabled. If the remote server requires it, you'll see this error. Enable TLS 1.2 in Schannel via registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Protocols\TLS 1.2\Client
DisabledByDefault = 0 (DWORD)
Enabled = 1 (DWORD)

2. Examine the Certificate Chain

Had a case where the server sent an intermediate CA certificate that was too large. Wireshark showed the TCP packets fragmenting the TLS handshake. Fix: Have the server admins install the intermediate CA on the server, or reduce the certificate chain size.

3. Firewall or Proxy Interference

Some firewalls (looking at you, older Cisco ASAs) do packet inspection on TLS handshakes and split them into chunks. If the error happens only from one network segment, check the firewall logs for TCP window scaling issues. Disable TCP window scaling on the client as a test:

netsh int tcp set global autotuninglevel=disabled

Then reboot. If it fixes it, you know the network gear is the culprit. Re-enable autotuning after: netsh int tcp set global autotuninglevel=normal

4. Update Windows

Microsoft fixed some Schannel timeout bugs in KB4489885 (Windows 10 1809) and later. If you're on an old build, get current. Seriously — I've seen this error vanish after a simple cumulative update.

What to Check If It Still Fails

If you've done all the above and it's still happening, capture a network trace. Use Wireshark on the client, filter for the server IP, and look for TCP retransmissions or zero-window events. The real fix is almost always one of these:

• Increase the Schannel timeout (most common)
• Enable TLS 1.2
• Fix the server certificate chain
• Update Windows

Don't waste time reinstalling apps or chasing phantom antivirus blocks. This is Schannel being impatient. Give it time, and it'll work.