Elasticsearch shard allocation failure fix

Cause 1: Disk space is too low (most common)

This one trips up everyone. When Elasticsearch nodes run out of disk space, the cluster stops allocating shards. It's the number one reason you see unassigned shards in GET _cluster/health with yellow or red status.

Elasticsearch has three disk-based watermarks. By default:

• Low watermark: 85% disk used — starts moving shards away.
• High watermark: 90% — stops allocating new shards.
• Flood watermark: 95% — puts the index in read-only mode.

If a node hits the flood watermark, your index becomes read-only. You'll see errors like FORBIDDEN/12/index read-only / allow delete (api). This is the cluster protecting itself.

The fix

First, free up disk space. Delete old indices that you don't need. Use DELETE /old-index-name. Or force-merge and shrink the index. But the quickest fix is to allow the index to be writable again:

PUT /your-index/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Then check the cluster health:

GET _cluster/health

If the status is still yellow, you need to reroute the unassigned shards manually:

POST /_cluster/reroute?retry_failed=true

This forces Elasticsearch to retry shard allocation. It works 80% of the time.

If you're hitting this regularly, raise the watermarks. On a production cluster, I set low to 80%, high to 85%, and flood to 90%. Use this:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "80%",
    "cluster.routing.allocation.disk.watermark.high": "85%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "90%"
  }
}

Also, check if you have a node with really low disk. Sometimes one node fills up while others have space. Elasticsearch won't move shards off that node if it's the only copy. Add more nodes or delete data from that node.

Cause 2: Cluster routing allocation is disabled

This one is sneaky. Someone (maybe you, maybe automation) turned off shard allocation. This happens during rolling restarts to prevent shards moving while a node is down. But if you forget to turn it back on, shards stay unassigned.

Check the setting:

GET _cluster/settings?pretty

Look for cluster.routing.allocation.enable. If it's set to none or primaries, you found the problem.

The fix

Enable allocation for all shards:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "all"
  }
}

Then run the reroute command again to force a retry:

POST /_cluster/reroute?retry_failed=true

In rare cases, this setting is set to new_primaries which only allows allocation for new primary shards. Same fix — set it to all.

Also, check if you have a cluster.routing.allocation.include._tier_preference or exclude setting that filters out nodes. For example, if you have cluster.routing.allocation.exclude._ip set to a node IP, that node won't get shards. Remove it:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.exclude._ip": null
  }
}

This is especially common in Elastic Cloud or Kubernetes environments where IPs change.

Cause 3: Node has the wrong role or is unavailable

Sometimes shards can't allocate because the node that holds them is gone. This happens when:

• A data node crashed and didn't come back.
• The node's role changed from data to master or ingest.
• You removed a node permanently but the shards didn't move in time.

You can see which node is missing by running:

GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason

Look for the unassigned.reason column. Common reasons are NODE_LEFT or EXISTING_SHARDS.

The fix

If the node is temporarily down (e.g., restarting), wait for it to come back. Then run the reroute command.

If the node is permanently gone, you need to reassign the shards elsewhere. Use the cluster reroute API to allocate them to a different node:

POST /_cluster/reroute
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "your-index",
        "shard": 0,
        "node": "new-data-node",
        "accept_data_loss": true
      }
    }
  ]
}

Warning: This accepts data loss. Only do this if you can't recover the original node or if the shard data is not critical (e.g., reindex from a source). I've used this many times when a node died with no backup. It's not ideal, but it gets the cluster green.

If you have replicas, you can allocate a replica from a primary that's still on another node:

POST /_cluster/reroute
{
  "commands": [
    {
      "allocate_replica": {
        "index": "your-index",
        "shard": 1,
        "node": "new-data-node"
      }
    }
  ]
}

Also check if the node role is correct. In elasticsearch.yml, the node.roles must include data. If it doesn't, shards won't land there. Edit the config and restart the node.

Quick-reference summary table

These three causes cover about 95% of shard allocation failures I've seen in production. Start with disk space, it's the most common. Then check settings. Then look at nodes. The reroute command with retry_failed=true is your best friend — use it after every fix.