Elasticsearch shard allocation failure: no enough nodes or disk

Quick answer

Run GET _cluster/allocation/explain to see why a shard is unassigned, then fix the cause: add nodes, free disk, or use cluster reroute to force allocate.

What's happening and why

When you see red or yellow cluster health in Elasticsearch, some shards are unassigned — they have no node to live on. The most common reason is that you don't have enough nodes to satisfy the replica allocation, or a node dropped offline and the shard replicas can't be placed. Another big one: disk-based allocation deciders block the shard because no node has enough free disk space.

What's actually happening here is that Elasticsearch's allocation deciders run a set of rules before placing a shard. If any rule fails — like the disk_watermark or the filter for node attributes — the shard stays unassigned. The cluster reroute API can override these deciders temporarily, but you should fix the root cause.

Fix steps

1. Check cluster health — run GET _cluster/health. Look at status. If it's red, you have unassigned primary shards. If yellow, only replicas are missing.
2. Find the unassigned shards — run GET _cat/shards?v. Shards with UNASSIGNED in the state column are the problem.
3. Ask Elasticsearch why — run GET _cluster/allocation/explain. This returns a JSON with deciders that tell you exactly why each shard can't be allocated. Look for disk_watermark or filter or awareness reasons.
4. Fix the cause —
   • If reason is disk_watermark: free disk on nodes, or raise the watermark thresholds temporarily with PUT _cluster/settings setting cluster.routing.allocation.disk.watermark.low to 90% or higher.
   • If reason is filter: your node has a exclude or require attribute that blocks the shard. Check index.routing.allocation.require and exclude settings on the index and node attributes.
   • If reason is awareness: you need to add nodes to satisfy awareness zones (e.g., you have 2 zones but only 1 zone has nodes up).
5. Force allocate shard as last resort — if you're sure the shard can go on a specific node, use the cluster reroute API:

   POST _cluster/reroute
   {
     "commands": [
       {
         "allocate_stale_primary": {
           "index": "your-index-name",
           "shard": 0,
           "node": "your-node-name",
           "accept_data_loss": true
         }
       }
     ]
   }

   Use allocate_stale_primary if the primary shard is unassigned (data might be stale). For replicas, use allocate_replica. For empty new indices, use allocate_empty_primary. This can cause data loss — only do this if you can't recover the original node.

Alternative fixes if main fails

• Restart nodes — sometimes a restart triggers cluster rebalancing and shards get allocated. Do a rolling restart: stop one node, wait for green, then next.
• Reroute with delay — if the node is temporarily down, set index.unassigned.node_left.delayed_timeout to something like 5m (5 minutes) to give it time to come back before you manually move shards.
• Delete the index and reindex — if the data is not critical, delete the index and reindex from a backup or source. This is the nuclear option but works.
• Check version mismatch — if nodes have different Elasticsearch versions, allocation might fail. All nodes must be compatible (usually within one minor version).

Prevention tip

Set up disk-based allocation watermarks before you run out of space. Use these cluster settings:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}

Also, always run at least 2 nodes for production — that way if one node goes down, replicas can still be allocated. Use index.number_of_replicas: 1 as a minimum.

The real fix is to understand why the shard can't be placed, not to force assign blindly. The allocation explain API is your best friend — use it before touching the reroute API.