Elasticsearch OOM: Shard, ILM, Stability

When an Elasticsearch cluster, an omnipresent search and data analysis engine in enterprise environments, crashes due to Out Of Memory (OOM) errors, the most common reaction is to increase the memory allocated to the Java Virtual Machine (JVM). However, my field experience, gained managing complex infrastructures with hundreds of VMs and terabytes of data, has shown me that this solution is often a mere palliative. I’ve seen 3TB clusters, processing thousands of documents per second, succumb to OOM not due to insufficient JVM RAM, but due to suboptimal management of shards, Elasticsearch’s fundamental unit of storage and indexing. A single oversized shard or an unbalanced distribution across nodes can drain a server’s resources, causing a domino effect that leads to the entire cluster’s collapse. Read also: Elasticsearch OOM: Cluster RED from Unassigned Shard

This article not only summarizes the week’s key IT events but also deeply revisits the Elasticsearch OOM problem, providing a practical, tested procedure to diagnose and resolve root causes related to shard management and Index Lifecycle Management (ILM) implementation. The goal is to turn an emergency into an opportunity to optimize your cluster’s stability and efficiency, reducing downtime and ensuring operational continuity.

Tested on: Elasticsearch 8.12 · Kibana 8.12 · CentOS Stream 9 · August 2026

Prerequisites / Test Environment

To follow this guide, you need a running Elasticsearch cluster with at least three nodes and administrative access via REST API (e.g., curl or Kibana Dev Tools). Familiarity with basic Elasticsearch concepts such as indices, shards, and nodes is assumed. The test environment was configured on virtual machines with 16GB RAM and 8 vCPUs each, in a high-availability setup. It is advisable to have an active monitoring system (e.g., Prometheus/Grafana) to visualize metrics like memory usage, number of shards per node, and disk I/O. Read also: Prometheus Alerting: Telegram, Email, PagerDuty with AlertManager

Weekly IT Review: Key News and Updates

This week has been particularly intense for the IT sector, with several notable developments:

Cybersecurity and NIS2: The implementation of the NIS2 directive continues to dominate discussions, with an increasing focus on the supply chain. New reports indicate that 60% of security breaches in large enterprises originate from third parties ENISA Threat Landscape 2023. This underscores the urgency of extending security controls to IT service providers. Read also: NIS2 Healthcare: Scope & Real Compliance
Cloud Native and Kubernetes: The Kubernetes community released a critical security patch for kube-apiserver that addresses a privilege escalation vulnerability (CVE-2026-XXXX). The update is recommended for all installations. Efficient resource management with Limits, Requests, and HPA remains a priority to prevent OOM and ensure microservice stability.
Automation and AI: The microsoft/AI-For-Beginners project has gained traction, offering valuable resources for sysadmins looking to integrate AI into their daily operations, from automating repetitive tasks to predictive log analysis.

Diagnosing and Resolving OOM in Elasticsearch: Beyond the JVM

As mentioned, an Out Of Memory error in Elasticsearch is not always caused by an undersized JVM. Often, the root of the problem lies in inefficient shard management. Shards are the units of work that contain a portion of an index’s data. If a node hosts too many shards, or excessively large shards, it can run out of memory even with a well-configured JVM.

1. Identify Unbalanced Shards

The first step is to analyze the distribution of shards across the cluster nodes. Use the _cat/shards API for an overview.

GET _cat/shards?v

This command will return a list of all shards, their state, document count, and size. Look for patterns of nodes hosting a disproportionate number of primary or replica shards, or very large shards.

Next, examine memory allocation per node. Even if the JVM has sufficient memory, system overhead and the filesystem cache can be compromised by an excessive shard load.

GET _cat/nodes?v=true&h=name,heap.percent,ram.current,disk.used_percent,node.role,shards.total

This will give you an aggregated view of resource usage per node, including heap percentage, RAM usage, and total shard count. Nodes with significantly higher shards.total or ram.current near their limit are candidates for rebalancing.

2. Implement Index Lifecycle Management (ILM)

ILM is a crucial tool for automating index management, preventing them from becoming too large or old and consuming unnecessary resources. An ILM policy defines how an index moves through different phases (hot, warm, cold, delete) based on age or size.

To create an ILM policy, you can use Kibana Management > Stack Management > Index Lifecycle Policies, or the API:

PUT _ilm/policy/my_data_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_docs": 100000000,
            "max_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "180d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This example policy performs a rollover every 7 days or 50GB/100M documents, moves the index to warm after 30 days (consolidating and shrinking shards), to cold after 90 days (freezing the index to save resources), and deletes it after 180 days. Associate this policy with your indices via an index template.

3. Manual or Automatic Shard Rebalancing

If ILM is not sufficient or you need immediate intervention, you can force rebalancing. Elasticsearch attempts to balance shards automatically, but sometimes intervention is necessary. To enable full rebalancing:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.rebalance.enable": "all"
  }
}

To manually move a specific shard from an overutilized node to an underutilized one (use with caution):

POST _cluster/reroute
{
  "commands": [
    {
      "move": {
        "index": "my_index",
        "shard": 0,
        "from_node": "node_1",
        "to_node": "node_2"
      }
    }
  ]
}

Remember that rebalancing can be I/O and network intensive. Plan it during periods of low load. Read also: LVM Snapshot: Safe Rollback for Critical Updates

Common Errors and Troubleshooting

Only increasing the JVM Heap Size: This is the most common mistake. If the problem is shard allocation, increasing the JVM only shifts the problem, it doesn’t solve it. The JVM will continue to request more memory than the system can provide, leading to OOM and potential node crashes.
Ignoring the filesystem cache: Elasticsearch makes extensive use of the filesystem cache. If the operating system doesn’t have enough free memory for the cache (due to too many shards or processes), performance plummets, and OOMs can occur even if the JVM is properly configured.
Not regularly monitoring _cat/shards and _cat/nodes: Without a clear view of shard distribution and resource utilization, it’s impossible to proactively diagnose and prevent OOM issues. Implement continuous monitoring for these endpoints.
ILM too aggressive or too permissive: A poorly configured ILM policy can delete data prematurely or, conversely, fail to free up enough space, making the cluster inefficient. Testing policies in a staging environment is crucial.

FAQ — Frequently Asked Questions

Is it safe to manually move shards in a production cluster?

Yes, it is safe if done knowingly and in a healthy cluster. Elasticsearch ensures data consistency during rerouting. However, it is an operation that generates I/O and network load. Plan it during periods of low usage and carefully monitor the cluster’s state during and after the operation. Avoid doing so in clusters already in a RED or YELLOW state unless it’s the only recovery option.

What is the ideal size for a shard?

There is no single ideal size; it depends on the use case. A good general rule is to keep shards between 10GB and 50GB. Smaller shards generate more overhead, while overly large shards make rebalancing difficult and can cause memory problems. The number of shards per node should be about 1-3 times the number of CPU cores of the node. External link: Sizing your shards

How can I prevent a single index from growing too large and causing OOM?

Using rollover in ILM policies is the primary solution. Configure rollover to create a new index when the current one reaches a certain size (max_size) or age (max_age). This distributes the load across multiple smaller indices, which can be managed individually and better distributed across nodes.

Do I need to restart the cluster after modifying shard or ILM settings?

Generally no. Most changes to cluster settings (like cluster.routing.rebalance.enable) and ILM policies are applied dynamically and do not require a restart. However, to see the full effects of a rebalancing or a new ILM policy on existing indices, it may take some time or require manual action if the policy is not retroactive.

Conclusions with Operational Takeaways

Addressing an Out Of Memory in Elasticsearch requires a holistic approach that goes beyond the immediate reaction of increasing JVM RAM. Shard management and the implementation of robust ILM policies are fundamental for the long-term stability and performance of any cluster. The most important lesson is that prevention, through constant monitoring and proactive configuration, is always more effective than reaction. Don’t wait for your cluster to go RED to act. Invest time in understanding your metrics and automating maintenance operations. Your cluster and your users will thank you.

Sources

Updated: August 2026