Cassandra Read Timeout During High Load Fix

When It Happens

You're running a Cassandra cluster — say 3 nodes on AWS m5.large instances, replication factor 3, consistency level QUORUM. Everything's fine until Black Friday or a marketing blast hits. Then your app starts logging "Operation timed out - received only 0 responses". Users see spinning spinners or error pages. The culprit here is almost always the coordinator node getting swamped.

Root Cause

Cassandra's coordinator handles routing requests to replicas, coordinating responses, and managing timeouts. Under load, the coordinator's thread pool fills up. Requests queue. Timeouts happen. The default read_request_timeout_in_ms is 5000ms (5 seconds). That's generous for normal use but tight when queues form. Also, if your consistency level is QUORUM or ALL, the coordinator must hear back from enough replicas before replying. Slow replicas or network hiccups add delay.

Don't bother tweaking phi_convict_threshold first — that's for failure detection, not read timeouts. The real fix is adjusting timeouts and reducing coordinator load.

Step-by-Step Fix

Check current timeout values — on any node run
```
nodetool gettimeout read
```
Default is 5000. If you see a lower number, someone already changed it.
Increase read timeout — edit cassandra.yaml on all nodes. Find
```
read_request_timeout_in_ms: 5000
```
Bump it to 15000 or 20000. Save and restart each node one by one. For example:
```
sudo systemctl restart cassandra
```
Boost coordinator concurrency — in the same file, increase
```
native_transport_max_threads: 128
```
to 256 or 512. This gives more threads to handle client connections. Save and restart again.
Lower consistency level if possible — if your app can tolerate stale data, change from QUORUM to LOCAL_QUORUM (if multi-datacenter) or ONE. In your driver code, set:
```
ConsistencyLevel level = ConsistencyLevel.LOCAL_ONE
```
Monitor heap usage — run
```
nodetool info
```
Check the "Heap Memory" line. If it's above 75%, increase heap size in cassandra-env.sh by changing
```
MAX_HEAP_SIZE="4G"
```
to 8G. Don't exceed 50% of physical RAM.

Still Failing? Check These

Garbage collection — long GC pauses cause timeouts. Check cassandra.log for GC events. If you see STW pauses over 1 second, switch to G1GC or ZGC.
Network latency — run
```
nodetool gossipinfo
```
If RTT values are over 10ms between nodes, your cluster is spread too thin. Move nodes closer together.
Hot partitions — if one key gets hammered, it can bottleneck. Run
```
nodetool tablehistograms keyspace_name table_name
```
If SSTables column shows high read counts for a single range, split that partition.
Compaction backpressure — heavy compactions steal resources. Check nodetool compactionstats. If pending compactions pile up, increase concurrent_compactors in cassandra.yaml.

"Most people overcomplicate this. 90% of read timeouts under load are fixed by bumping timeouts and adding threads. Only if that fails do you need to dive into GC or partitioning."