Kubernetes Production: First Deploy, Errors, and Fixes

Q: What is the difference between livenessProbe and readinessProbe?

livenessProbe tells Kubernetes if the container is "alive" and functioning. If it fails, Kubernetes restarts the container. readinessProbe, on the other hand, indicates whether the container is ready to serve traffic. If it fails, Kubernetes stops sending traffic to that pod but does not restart it. They are complementary to ensure both stability and service availability.

Q: What to do when a pod is in CrashLoopBackOff?

The first thing is to use kubectl describe pod to see recent events and kubectl logs for container logs. Often the cause is an application that is not starting correctly, an internal crash, or a misconfigured livenessProbe. Also, check the controller manager or kubelet logs for more system-level errors.

Q: How to prevent OOMKilled in containers?

Primarily, by defining resources.limits.memory for each container. Ensure the limit is sufficient for the application but not so high as to allow excessive consumption. Closely monitor memory usage and optimize your applications to reduce their footprint. Also, consider using requests.memory to ensure the pod is only scheduled on nodes with sufficient memory.

Friday afternoon. Kubernetes in production for 6 hours. 40 pods in CrashLoopBackOff. The CEO waiting for the demo. The team not understanding what was happening. We had forgotten to configure liveness probes.

The first deployment of a Kubernetes cluster in production is a rite of passage for every DevOps team or SysAdmin. It’s a moment of high anticipation, but also of potential disaster. In my professional journey, I have led many complex migrations, but few events have left an impression like our first true approach to Kubernetes in an enterprise environment. That day taught us invaluable lessons, forging a more robust and conscious approach to container orchestration. If you are about to face your first deployment, or if you have already experienced similar moments, this experience will offer practical insights to avoid common pitfalls and quickly stabilize your environment.

Prerequisites / Test Environment

Our scenario involved migrating a suite of monolithic applications, previously managed with Docker Compose, to a new Kubernetes cluster. The goal was to improve scalability, resilience, and accelerate the continuous deployment process. The environment consisted of 5 worker nodes and 3 master nodes, with Ceph storage and an external load balancer. The applications were primarily Java Spring Boot, Node.js, and some Python microservices. The migration plan was ambitious: three weeks to get everything online, including integration and performance testing.

The Context: Migrating from Docker Compose to Kubernetes in 3 Weeks

The decision to move to Kubernetes was driven by the need to manage a growing workload with unpredictable peaks and to standardize the infrastructure for future developments. Docker Compose had reached its limits in terms of orchestration and auto-healing. Kubernetes promised a solution to many of our operational problems. We dedicated weeks to team training, application containerization, and defining YAML manifests. The project was on track, with a staging environment that seemed stable and performant. Pressure was high, given the aggressive timeline and the strategic importance of the migration for the company. A 2023 Cloud Native Computing Foundation (CNCF) survey revealed that 96% of organizations use or plan to use Kubernetes, highlighting its centrality but also the complexity of the initial approach.

Hours 0-2: Deployment Successful, Everything Seems to Work

The production deployment day began with euphoria. After applying the manifests, kubectl get pods showed all pods in a Running state. Services responded correctly, traffic was routed without issues, and initial functional checks were positive. The entire team breathed a sigh of relief. The CEO had been informed of the initial success, and the demo was scheduled for the afternoon, a crucial moment to showcase the value of the new infrastructure. Basic cluster metrics, such as CPU and memory usage, seemed normal. At that point, we thought we had passed the most critical phase.

Hours 2-6: CrashLoopBackOff on Half the Pods, Unknown Cause

The euphoria was short-lived. About two hours after deployment, monitoring began to signal anomalies. A growing number of pods entered the CrashLoopBackOff state. Initially one or two, then ten, twenty, up to almost half of the 80 total pods. Application logs were sparse or nonexistent. kubectl logs didn’t help much, only showing continuous container restarts. The situation was critical: the CEO’s demo was about to begin, and the service was partially degraded, with a direct impact on users. The pressure became unbearable.

To investigate, the first step was to use kubectl describe to get a detailed overview of the problematic pods’ status:

kubectl describe pod <pod-name> | grep -A20 Events

This output showed a series of Liveness probe failed and Restarting container events. It wasn’t a direct application error, but a problem with Kubernetes’ container lifecycle management.

The Error: Liveness Probe with Too Low Timeout on Java Application

The main root cause was identified in livenessProbe configurations that were too aggressive for our Java Spring Boot applications. Java applications, especially during startup, require some time to initialize the Spring context, load dependencies, and open database connections. Our livenessProbe was configured with an initialDelaySeconds that was too low (e.g., 5 seconds) and an equally restrictive timeoutSeconds (e.g., 1 second). This caused Kubernetes to consider the container unresponsive and restart it, again and again, before the application had a chance to fully start up. A vicious cycle of continuous restarts was created, preventing pods from reaching a stable Running state.

The immediate solution was to significantly increase initialDelaySeconds and timeoutSeconds to allow Java applications to complete their startup. A more realistic configuration for a Java Spring Boot application might be:

livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 10
  failureThreshold: 3

This gives the application 60 seconds to start before the first check and allows up to 3 consecutive failures before declaring the pod unhealthy, with a 10-second interval between checks. This is a classic error, but easy to make under pressure. According to a 2024 Datadog report, 30% of production incidents on Kubernetes are related to incorrect probe configurations.

The Second Problem: No Resource Limits Configured

While we were solving the liveness probe issue, another critical point emerged. Some worker nodes showed excessive memory and CPU usage, with sudden performance drops. Inspecting the pods on those nodes, we noticed that we had not configured resources.limits in the manifests. Our applications, although optimized, had no upper limit on resource consumption. In a production environment, this is a serious error that can lead to instability and cascading effects.

Without resource limits, a misbehaving application (e.g., with a memory leak) can consume all available resources on the node, starving other pods and bringing the node itself to an unstable state. In such cases, Kubernetes intervenes by terminating processes that exceed available resources, often with an OOMKilled (Out Of Memory Killed).

Hours 6-24: Cascading OOMKilled Across the Node

The lack of resource limits and the initial liveness probe problem created a perfect storm. After correcting the probes, applications began to start, but without limits, some began to consume more memory than expected, triggering cascading OOMKilled events. A worker node with 32GB of RAM found itself with several Java pods that, without limits, tried to allocate more memory than necessary. The Linux kernel intervened, terminating processes to protect the operating system. This led to further pod restarts and, in some cases, the entire node becoming unresponsive. To check the OOMKilled status of a pod, you can use:

kubectl get pod <pod-name> -o json | jq '.status.containerStatuses[].lastState'

This command shows the last state of the container, and if reason: OOMKilled is found, the problem is confirmed. This scenario is particularly insidious because it can be difficult to diagnose without adequate monitoring and a deep understanding of how Kubernetes and the Linux kernel manage resources.

How We Recovered and Stabilized the Cluster

Recovery required a multi-phase, coordinated approach:

Correcting Liveness/Readiness Probes: We patched all deployments to update probe configurations, increasing initialDelaySeconds and timeoutSeconds, especially for Java applications. We also introduced separate readinessProbe to indicate when the application was ready to receive traffic, distinguishing it from the mere ‘liveness’ of the pod. This drastically reduced CrashLoopBackOff events.

Setting Resource Limits: We defined resources.requests and resources.limits for each container. Initially, we opted for conservative limits based on observations in staging, with the intention of refining them later with more precise load tests. This prevented a single pod from starving the entire node.

Horizontal and Vertical Scaling: We temporarily scaled the number of worker nodes to better distribute the load and provide a resource buffer. At the same time, we increased the resources (RAM/CPU) on existing nodes to better handle peaks. This gave us time to analyze and optimize.

Advanced Monitoring: We configured more granular monitoring dashboards with Prometheus and Grafana, focusing on key metrics such as CPU, memory, disk I/O usage for each pod and node, and pod states. This allowed us to quickly identify new anomalies.

Communication and Rollback Plan: We communicated constantly with the CEO and stakeholders, explaining the situation and the steps for resolution. Fortunately, we had a rollback plan (though not thoroughly tested) that would have allowed us to revert to Docker Compose, but we managed to stabilize the cluster before needing to invoke it.

Common Errors and Troubleshooting

This incident highlighted several common errors in Kubernetes deployment:

Misconfigured Liveness/Readiness Probes: Too aggressive, not accounting for application startup times (especially Java). The official Kubernetes documentation on probes is an excellent starting point for configuring them correctly. External link: Kubernetes Documentation on Probes
Lack of Resource Limits: Leaving containers without CPU and memory limits is a recipe for disaster in production, leading to OOMKilled and node instability.
Insufficient Load Testing: The staging environment does not always accurately replicate production load and traffic patterns. It is essential to perform realistic load tests.
Weak Monitoring: Without robust monitoring and timely alerts, it is difficult to diagnose and react quickly to incidents.
Untested Rollback Plans: Having a plan is good, testing it is crucial. You cannot assume it will work under stress.

FAQ — Frequently Asked Questions

What is the difference between livenessProbe and readinessProbe?

livenessProbe tells Kubernetes if the container is “alive” and functioning. If it fails, Kubernetes restarts the container. readinessProbe, on the other hand, indicates whether the container is ready to serve traffic. If it fails, Kubernetes stops sending traffic to that pod but does not restart it. They are complementary to ensure both stability and service availability.

How can I correctly set resource limits for my applications?

The best way is to monitor your applications’ resource consumption in a staging environment under typical load. Start with requests slightly below average consumption and limits slightly above the detected peak consumption. Refine the values over time, based on metrics and stress tests. A good starting point is to set requests.cpu and requests.memory to ensure fair scheduling, and limits.memory to prevent OOMKilled.

What to do when a pod is in CrashLoopBackOff?

The first thing is to use kubectl describe pod to see recent events and kubectl logs for container logs. Often the cause is an application that is not starting correctly, an internal crash, or a misconfigured livenessProbe. Also, check the controller manager or kubelet logs for more system-level errors.

How to prevent OOMKilled in containers?

Primarily, by defining resources.limits.memory for each container. Ensure the limit is sufficient for the application but not so high as to allow excessive consumption. Closely monitor memory usage and optimize your applications to reduce their footprint. Also, consider using requests.memory to ensure the pod is only scheduled on nodes with sufficient memory.

Conclusions with Operational Takeaways

Our first Kubernetes production deployment was a rich learning experience, riddled with common but solvable errors. The most important lesson is that theory and practice can diverge significantly, especially in complex environments. The accurate definition of livenessProbe and readinessProbe, coupled with meticulous configuration of resource limits, are fundamental pillars for the stability of any Kubernetes cluster. Never underestimate the importance of realistic load tests and proactive monitoring. This reactive approach allowed us to stabilize a critical service in a few hours, learning to manage pressure and quickly identify culprits. With adequate preparation and a robust troubleshooting methodology, it is possible to turn a potential disaster into an opportunity for continuous growth and improvement.

5 Things to Do BEFORE Your First Production Deployment

Test Probes in Staging: Simulate your applications’ startup with the livenessProbe and readinessProbe configurations you will use in production. Ensure startup times are handled correctly.
Define Realistic Resource Limits: Do not go into production without requests and limits for CPU and memory. Monitor your applications and set values based on real data.
Perform Load and Stress Tests: Don’t rely solely on functional tests. Simulate production load (and beyond) to see how the cluster reacts and where bottlenecks are.
Configure Monitoring and Alerting: Implement a comprehensive monitoring system (e.g., Prometheus/Grafana) with alerts configured for critical pod states, high resource usage, and system errors. Good monitoring reduces MTTR (Mean Time To Resolution).
Prepare a Rollback Plan: Even if you hope not to use it, having a detailed plan to revert to the previous configuration (e.g., Docker Compose, or an earlier version of the manifests) is crucial. And, most importantly, test it!

Last updated: July 2026