Yeah, that E11000 on a sharded cluster with a partial unique index is a special kind of headache. You're not alone—it's one of those errors that makes you question every decision you've made with your database. Let's get to the fix first, then we'll talk about why your setup is fighting you.
The Fix: Align the Partial Filter With the Shard Key
The most common cause of E11000 in this scenario is that your partial index has a filter expression that doesn't include the shard key. MongoDB requires that any unique index on a sharded collection include the shard key as a prefix—and partial indexes are no exception. If the filter doesn't reference the shard key, the index can't be enforced correctly across chunks, and you'll get spurious duplicate key errors even when the data looks unique.
Here's what you do:
- Check your shard key:
sh.status()shows it. Note it down. - Look at your partial index definition. It probably looks like
{ "email": 1 }with a filter like{ "email": { $exists: true } }. - Rebuild the index with the shard key included as a prefix and the filter referencing that shard key. For example, if your shard key is
{ "tenantId": 1 }, your index should be{ "tenantId": 1, "email": 1 }with filter{ "tenantId": { $exists: true }, "email": { $exists: true } }.
db.collection.dropIndex({ "email": 1 })
db.collection.createIndex(
{ "tenantId": 1, "email": 1 },
{ unique: true, partialFilterExpression: { "tenantId": { $exists: true }, "email": { $exists: true } } }
)If the index already exists but is wrong, drop it and recreate. That's it. The error will stop.
Why This Works
What's actually happening here is that MongoDB enforces unique constraints per chunk on a sharded cluster. Each chunk is owned by a single shard, and the shard checks uniqueness within its own data. But for the constraint to be global, the index has to be able to route a document to the right shard based on the unique key. If the unique key doesn't include the shard key, there's no way to guarantee that two documents with the same key value end up on the same shard—so MongoDB has to do extra checks that can produce false positives.
By including the shard key as a prefix, you're telling MongoDB: "Uniqueness only needs to be enforced within the same shard key value." That's a valid constraint and it can be enforced locally. The partial filter also needs to match the shard key because the index is only consulted for documents that pass the filter. If the filter doesn't reference the shard key, the index might not be used for documents that span chunks, causing the duplicate key error.
Also, you need to be aware that when you have a partial index, the filter expression is evaluated against the document. If a document doesn't match the filter, it's not in the index—so the unique constraint doesn't apply to it. That's fine, but it means your filter has to be precise. If you're seeing E11000 on documents that shouldn't even be indexed, the filter is likely too narrow or too broad.
Less Common Variations
1. The Filter Uses a Field That's Not in the Index
Sometimes the filter references a field that's not part of the index keys. MongoDB allows that, but it can cause weird behavior because the index can only be used for queries that match the filter. If you're trying to upsert a document that doesn't match the filter, you'll get E11000 because the unique constraint is applied anyway (the index is considered, but the document isn't in it). The fix here is to make sure your application logic only upserts documents that satisfy the filter, or adjust the filter to match reality.
2. Collation Differences
If you created the collection with a specific collation (like strength 2 for case-insensitive), and your index has a different collation, uniqueness is evaluated differently. On a sharded cluster, this can cause E11000 when two documents differ only in case. The fix is to rebuild the index with the same collation as the collection, or drop the collation entirely if you don't need it.
3. Index Build in Progress
If you're building a partial unique index on a sharded cluster, you might see E11000 during the build itself. That's because data is being migrated between chunks, and the index is being built incrementally. The error can be transient. Wait for the build to finish, then check again. If it persists, you have actual duplicates—query for them with aggregation on the indexed fields.
4. Shard Key Changed After Index Creation
You can't change a shard key, but if you inherited a poorly designed cluster, the index might have been created when the shard key was different. That's a legacy mess. You'd need to dump and reimport the data with the correct index. No way around it.
Prevention
The real fix is to design your partial unique indexes with the shard key in mind from day one. Before you create any unique index on a sharded collection, ask yourself: "Does this index include the shard key as a prefix?" If not, stop and rethink. It's not a suggestion—it's a hard requirement.
Also, test your partial filter with real data. Write a script that inserts documents that match and don't match the filter, and make sure the unique constraint behaves as expected. MongoDB doesn't validate filter logic; it just applies it. Your job is to ensure the filter is exactly what you want.
When you create the index, use background: true in older versions (pre-4.2) to avoid blocking writes, but in modern versions (4.2+) index builds are hybrid and won't block. Just be aware that building a unique index on a large collection will fail if duplicates exist. Run a duplicate check first:
db.collection.aggregate([
{ $group: { _id: { field1: "$field1", field2: "$field2" }, count: { $sum: 1 } } },
{ $match: { count: { $gt: 1 } } }
])If that returns results, clean them up before creating the index.
One more thing: if you're using MongoDB Atlas or a cloud provider, their support folks see this error daily. They'll tell you the same thing. So save yourself the ticket and fix the index design yourself. It's a 10-minute change that prevents hours of confusion.