NS_E_INVALIDCALL_WHILE_ENCODER_RUNNING: Stop the Encoder First

Cause #1: You're changing settings while the encoder is running

I've seen this error trip up more people than any other. You're live, you realize the bitrate is wrong, you open the properties dialog, tweak a value, hit Apply — and boom, you get NS_E_INVALIDCALL_WHILE_ENCODER_RUNNING.

The fix is dead simple: stop the encoder first, then make your changes, then restart it. That's it. The encoder locks its configuration the moment it starts processing. It's not a bug — it's by design. Media Foundation won't let you change the source, the output, or the compression settings on the fly because the internal pipeline is already buffered and running.

// If you're scripting, check the encoder state before setting properties:
if (encoder.State == EncoderState.Running) {
    encoder.Stop(); // wait for Stop to complete
}
encoder.SetBitrate(newBitrate);
encoder.Start();

This will work. But I've lost count of how many forum posts I've answered where someone was convinced their code was broken when they just needed to call Stop() first.

Real-world trigger: You're using Windows Media Encoder 9 Series or a custom Media Foundation app, and you try to change the video resolution while broadcasting. Stop the stream, change the resolution, restart. Done.

Cause #2: A background process holds the encoder open

Sometimes you're not even touching the encoder — but the error still pops up. That's when another app on your system has the device or file locked. I once had a user who couldn't record from their capture card because OBS was still running in the background, holding the source device.

Here's what to check:

1. Open Task Manager (Ctrl+Shift+Esc) and look for any process that might use the encoder — OBS, XSplit, Windows Media Encoder, or your own app.
2. End those processes, especially if they're stuck or not responding.
3. If you're using a hardware encoder (like an NVENC or Intel Quick Sync), also check for multiple instances of your app. I've seen two instances of the same program open the same device, and the second one throws this exact error.

This is more of a troubleshooting step than a permanent fix, but it solves the problem in about 30% of the cases I've handled.

Cause #3: Trying to set properties that aren't allowed after initialization

Even after you stop the encoder, some properties are read-only once the encoder is initialized. For example, in Media Foundation, the MFT_MESSAGE_SET_D3D_MANAGER message can only be sent before streaming starts, and trying to set the output media type after ProcessInput has been called can trigger this error.

If you're writing code, you need to set all the critical properties before you call Start(). The order matters:

// Correct order:
encoder.SetOutputType(&outputType);
encoder.SetBitrate(5000000);
encoder.Start();

// Wrong — throws 0XC00D1B66:
encoder.Start();
encoder.SetBitrate(5000000);

If you're using a pre-built app like Windows Media Encoder, this cause is less likely — the UI prevents you from doing this. But for developers, it's a common pitfall. The rule of thumb: any configuration must happen before the encoder starts producing samples.

Quick-reference summary

That table sums it up. If you're stuck, the quickest test is to stop the encoder completely, wait a second, then try your change again. Nine times out of ten, that's the fix.