In this lesson, you will learn how to correlate application-level Garbage Collection (GC) metrics with Kubernetes health check failures to prevent unnecessary pod restarts.
The "Stop-the-World" Conflict
In managed-memory runtimes like Java (JVM) or Go, the Garbage Collector occasionally performs "Stop-the-World" (STW) events. During these cycles, the runtime halts all application threads to reclaim memory. If your heap is misconfigured or your application is under high pressure, these pauses can extend beyond the timeoutSeconds defined in your livenessProbe.
When the kubelet attempts to check your service's health during a long GC pause, the application fails to respond in time. Kubernetes interprets this silence as an application crash, killing and restarting the pod. This creates a "death spiral": the pod restarts, attempts to warm up its cache, triggers a major GC cycle, hits a STW pause, fails the health check, and is killed again.
Monitoring the Pause
To debug this, you must look at runtime-specific metrics. For a JVM application, you should export GC statistics via JMX or Prometheus (e.g., using micrometer). You are looking for the jvm_gc_pause_seconds_max metric. If this value spikes above your livenessProbe threshold, you have found the culprit.
Consider this probe configuration:
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 2 # GC pause > 2s will trigger a restart
If your GC logs show frequent 3-second pauses, your timeoutSeconds is too aggressive. Instead of simply increasing the timeout, analyze the allocation rate. Are you over-provisioning memory, causing the GC to thrash, or is the heap too small for your baseline traffic? Use the metrics to adjust your heap settings (-Xmx) to ensure the GC frequency stays within the bounds of your probe configuration.
By aligning your liveness timeout with your observed maximum GC pause duration—while simultaneously tuning memory limits—you ensure that the orchestrator respects the realities of your runtime environment.
Key Takeaway: Monitor GC pause duration metrics and ensure your Kubernetes liveness probe timeout is sufficiently higher than the maximum observed STW pause to prevent false-positive pod restarts.