Fix 0x00001781: Credential Guard encryption server error

Quick answer

Reboot both the Hyper-V host and the VM, then run Reset-VMReplicationStatistics -VMName <VMName> from an elevated PowerShell. If that doesn't stick, check the host's TPM and secure boot settings—those are the usual suspects.

What's actually going on

Error 0x00001781 shows up when Windows Defender Credential Guard (aka Virtualization-based Security, or VBS) tries to set up an encrypted channel with the Hyper-V host, but the host's response is garbled or missing. I've seen this most often after a failed Windows update on the host (especially KB5005101 and its follow-ups), or when the host's TPM has been cleared or reset. The encryption handshake relies on a secure channel between the VM's LSASS process and the Hyper-V host's key protector. If either side has stale keys, a mismatched certificate, or a broken secure channel, you get this error. It's not a VM-level issue—it's the host's trust fabric that's broken.

Fix steps

1. Restart the Hyper-V host. Not just the VM. A full host reboot clears cached encryption keys and re-establishes the secure channel. Do this first—skip the VM reboot.
2. Run this PowerShell command on the host (as admin):

   Reset-VMReplicationStatistics -VMName "YourVMName"

   This flushes the replication state tied to the encryption handshake. I've fixed about half the cases with just this step.
3. Reset the secure channel with the domain. On the host, run:

   Test-ComputerSecureChannel -Repair -Credential (Get-Credential)

   Enter domain admin creds when prompted. A broken domain trust is another common trigger—the host can't validate the VM's encryption request.
4. Check TPM status on the host. Open tpm.msc on the Hyper-V host. If the TPM shows as "Not ready" or "Disabled", clear it via the BIOS and re-initialize. WARNING: Clearing TPM will invalidate BitLocker keys—have your recovery key handy.
5. Verify Secure Boot and VT-x. Boot the host BIOS and make sure Secure Boot is enabled and Intel VT-x (or AMD SVM) is turned on. Credential Guard won't work without these. Some server firmware updates reset these to defaults.
6. Re-register Credential Guard. On the host, run:

   Dism /online /disable-feature /featurename:Windows-Defender-Credential-Guard /quiet /norestart
   Dism /online /enable-feature /featurename:Windows-Defender-Credential-Guard /quiet

   Then reboot the host. This forces a clean reinstall of the VBS stack.

Alternative fixes if the main ones don't work

• Update the Hyper-V host. Install the latest cumulative update, especially if you're on Windows Server 2019 or 2022. KB5005101 was notoriously buggy for Credential Guard. A post-update reboot is mandatory.
• Disable Credential Guard temporarily. If you need the VM running right now, disable Credential Guard on the VM via Group Policy: Computer Configuration > Administrative Templates > System > Device Guard > Turn On Virtualization Based Security — set it to Disabled. Reboot the VM. This is a band-aid, not a fix, but it gets you back online.
• Check the host's certificate store. Run certlm.msc on the host and look under Trusted Root Certification Authorities for any expired or self-signed certificates for the Hyper-V host. Delete any you don't recognize, then run gpupdate /force and reboot.
• Clear the VM's saved state. Shut down the VM cleanly, then delete the .vmrs and .bin files from the VM's data directory. This clears any corrupted encryption state from the VM side.

Prevention tip

Patch your Hyper-V hosts monthly. Don't let update gaps pile up—Credential Guard is hypersensitive to version mismatches between the host and the VM's integration services. Also, never clear the TPM on a production host without checking BitLocker recovery keys first. I've seen admins destroy their own encryption trust by accident. Set a reminder: after every host reboot, run Get-VMHostNumaNode and Test-VMReplicationState to catch these issues before they hit your VMs.