When an Elasticsearch cluster manages terabytes of data, stability is paramount. A seemingly minor error can have enormous repercussions, blocking critical log ingestion and compromising an organization’s ability to monitor security and ensure compliance. In this article, I will recount a direct troubleshooting experience on a 3TB Elasticsearch cluster, where an apparent Out Of Memory (OOM) issue turned out to be something far more insidious: an unassigned shard due to insufficient disk space on a node. I will share the diagnosis process, the implemented solution, and the lessons learned to prevent similar incidents in the future.
Tested on: Elasticsearch 7.x · Ubuntu 22.04 LTS · July 2026
Prerequisites / Test Environment
To follow this analysis, familiarity with basic Elasticsearch concepts, including nodes, shards, replicas, and the REST API, is helpful. The reference environment is an Elasticsearch cluster distributed across multiple Linux nodes, with a continuous log ingestion load from hundreds of sources, totaling approximately 3TB. The specific Elasticsearch version was 7.17, but the principles also apply to more recent versions.
The Context: ELK Cluster with 3TB of Continuous Log Ingestion
I was working in an enterprise environment where an ELK (Elasticsearch, Logstash, Kibana) cluster was at the core of log collection and analysis. This system managed about 3TB of data distributed across several nodes, with a constant ingestion flow. Logs were crucial for the SOC (Security Operations Center) for threat detection and for compliance teams for auditing and reporting. The health of this cluster was a top priority, with a very stringent SLA (Service Level Agreement) for availability. Read also: Prometheus Alerting: Telegram, Email, PagerDuty with AlertManager
The Symptom: Cluster Health RED, All Searches Fail
One Monday morning, every sysadmin’s nightmare became a reality: the Elasticsearch cluster went into RED status. This means one or more primary shards were unassigned, making some or all indices inaccessible. The consequences were immediate and dramatic: Kibana dashboards showed errors, all searches failed, and, even more critically, new log ingestion was blocked, creating a significant backlog. The situation was critical, with the risk of losing valuable data for security analysis. 73% of breaches start with stolen credentials, but the inability to analyze logs in real-time makes detection impossible — Verizon DBIR 2025.
First, I checked the cluster status:
curl -X GET 'localhost:9200/_cluster/health?pretty'The output confirmed a status: red, indicating the presence of unassigned shards.
Diagnosis: A Single Index with Undistributed Shards
The key to resolving these issues is rapid and precise diagnosis. Instead of jumping to conclusions about memory or CPU problems, I used the Elasticsearch API to get a detailed explanation of shard allocation. This API is an incredibly powerful tool for understanding why a particular shard has not been allocated.
curl 'localhost:9200/_cluster/allocation/explain?pretty'The output of this command was illuminating. It pointed to a single primary shard of a daily index (logs-2026) that could not be allocated. The reason given was clear: the node is above the low watermark for disk usage. In other words, the node on which Elasticsearch was trying to allocate the shard had exceeded the disk usage threshold, preventing the operation.
In this specific case, it was a relatively small shard, about 4GB, on a node that had only 200MB of free space. This created a stalemate: the shard could not be allocated because there was no space, and without the primary shard, the index was unusable.
Why It Happened: Index with Replica on Full Disk Node
The root cause was twofold. Firstly, Elasticsearch’s default configuration provides one replica for each primary shard. This is an excellent practice for resilience, but in this scenario, the primary replica of our logs-2026 index was configured to be allocated to a specific node which, unfortunately, had almost run out of disk space. Read also: Oracle DBA Checklist: 10 Daily Checks for Production Databases (2026)
Secondly, disk space monitoring on the data nodes was not granular enough. We had generic alerts but not specific ones for Elasticsearch’s “low watermark” thresholds (typically 85%). This allowed the node to silently fill up to the critical point. Once the disk.watermark.low threshold was exceeded, Elasticsearch stopped allocating new shards to that node to prevent further problems, but it did not automatically move existing ones unless the disk.watermark.high threshold (90-95%) was reached.
How We Recovered Without Data Loss
Fortunately, recovering the cluster was faster than the diagnosis, and without any data loss, as the shard was simply “unassigned” and not corrupted. The immediate solution was to force the shard’s reallocation to a different node with sufficient disk space.
curl -X POST 'localhost:9200/_cluster/reroute' \n -H 'Content-Type: application/json' \n -d '{"commands":[{"allocate_stale_primary":{"index":"logs-2026","shard":0,"node":"node_with_space","accept_data_loss":false}}]}'CAUTION: The allocate_stale_primary command must be used with extreme care and only when you are certain that the existing primary shard is ‘stale’ (i.e., no longer valid or reachable) and there is no risk of data loss. In our case, the shard was simply unassigned, not corrupted. We verified that the original node was ‘down’ or unreachable for reallocation. It is essential to replace node_with_space with the actual name of an available node with sufficient disk space. After executing this command, the cluster quickly returned to GREEN status, and ingestion and searches resumed normal operation. Read also: HAProxy Load Balancing: High Availability for Web Apps (2026)
The Tuning We Should Have Done Immediately (Heap, Shard Sizing)
This experience highlighted the importance of proactive tuning. Two key areas we should have optimized immediately were heap memory and shard sizing.
Heap Memory Tuning (JVM)
The JVM heap memory is crucial for Elasticsearch performance. A general rule is to allocate about 50% of the node’s total RAM to the heap, but never exceed 32GB. This is because a heap larger than 32GB can trigger the use of compressed pointers, which reduces JVM efficiency. The configuration is found in the jvm.options file or directly in elasticsearch.yml:
# jvm.options (or elasticsearch.yml)
-Xms4g
-Xmx4g # Example for a node with 8GB RAMIn our case, the heap was configured correctly, but it is a fundamental aspect to always check.
Shard Sizing and Allocation
The size and number of shards directly affect stability and performance. Shards that are too large or too small can cause problems. A good practice is to keep shards between 10GB and 50GB. Furthermore, it is essential to constantly monitor the disk space of data nodes and configure adequate alarm thresholds for watermarks. 90% of performance incidents on Elasticsearch are related to incorrect shard or memory management (Elastic Blog, 2023).
3 Rules to Avoid Repeating the Error
To prevent a similar incident from recurring, we implemented the following rules:
- Granular Disk Space Monitoring: Implement monitoring that not only checks total available space but is also aware of Elasticsearch’s
disk.watermark.lowanddisk.watermark.high. Specific alerts must be generated when a node approaches these thresholds, allowing intervention before a blockage occurs. Read also: FortiGate Misconfiguration: 3 Hours of Open Firewall in Production (2026) - Automated Space Reprovisioning: Implement automatic or semi-automatic processes to free up disk space on data nodes (e.g., deleting old indices or moving data to long-term storage) when thresholds are exceeded. This will reduce reliance on manual intervention in emergency situations.
- Periodic Resilience Testing: Periodically perform resilience tests by simulating the loss of a node or disk filling on a node, to verify that the cluster recovers automatically or that alerts function as expected. This also includes verifying shard allocation and rebalancing policies.
Common Errors and Troubleshooting
One of the most common errors is confusing a RED status with a general performance or memory issue. As we’ve seen, the cause can be very specific, such as insufficient disk space. Another mistake is ignoring YELLOW status warnings, which indicate unassigned replica shards; these can degenerate into RED if the problem persists and the primary node fails. Always using _cluster/allocation/explain is the first step for accurate diagnosis.
FAQ — Frequently Asked Questions
What exactly does ‘cluster health RED’ mean?
Cluster health RED indicates that one or more primary shards of one or more indices are unassigned. This means that a portion of your data is inaccessible, and ingestion may be blocked. It is the most severe state for an Elasticsearch cluster and requires immediate action to restore full operation.
Can I lose data if the cluster is in RED status?
Generally no, if the problem is the non-allocation of an existing shard. The primary shard’s data is still present on the original node’s disk or in backups. However, if you do not intervene, you might lose data that is no longer being ingested while the cluster is blocked, or if the node hosting the primary shard suffers an irreversible failure.
How can I prevent data nodes from filling up?
It is crucial to implement proactive disk space monitoring and configure ILM (Index Lifecycle Management) policies to automatically manage index retention. This includes automatically deleting older indices or moving them to less expensive storage (like S3 or cold storage) when they are no longer needed for fast queries.
When should I use _cluster/reroute?
_cluster/reroute should only be used in emergency situations, after carefully diagnosing the cause of the problem and exhausting automatic options. Using accept_data_loss: true is extremely risky and should only be done as a last resort and with full awareness of the risk of data loss.
Conclusions with Operational Takeaways
The incident of the 3TB Elasticsearch cluster blocked by a single shard was a powerful reminder: even in the most complex environments, problems can arise from seemingly trivial causes. The key is to have the right tools for diagnosis (_cluster/allocation/explain), clear processes for recovery (_cluster/reroute), and, above all, a strong emphasis on prevention through granular monitoring and proactive configuration tuning. Do not wait for your cluster to go RED to discover its weaknesses. Invest in monitoring, training, and data management policies to ensure the resilience of your infrastructure.
Updated: July 2026
