Handling Graceful Connection Draining During Deployments

In this lesson, you will learn how to configure termination grace periods to ensure your Kubernetes pods finish processing active requests before they are removed from service during a deployment.

The Problem: The "Connection Reset" Race

When you perform a rolling update, Kubernetes sends a SIGTERM signal to your old pods and simultaneously removes them from the Service endpoints. However, there is a tiny window of time where the load balancer or the Service proxy might still be routing traffic to a pod that is already shutting down. If the pod closes connections immediately upon receiving SIGTERM, any in-flight requests will be abruptly dropped, resulting in "Connection Reset" errors for your users.

The Solution: Termination Grace Periods

To handle this gracefully, we need to ensure the pod stays alive long enough to finish its current work. Kubernetes provides the terminationGracePeriodSeconds field in your Pod specification to manage this lifecycle.

When a pod is terminated, it is given a grace period (defaulting to 30 seconds). During this time:

1. The pod receives SIGTERM.
2. The application should intercept this signal, stop accepting new connections, but keep processing existing ones.
3. Once the grace period expires, Kubernetes sends SIGKILL to force-stop the process.

You can adjust this in your deployment manifest:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: my-app
        image: my-app:v1

Best Practices for Draining

Simply extending the grace period isn't enough; your application code must be signal-aware. When your app receives SIGTERM, it should enter a "draining" state. In this state, it should:

• Stop listening for new incoming connections.
• Wait for existing request handlers to complete.
• Close database connections and clean up resources only after active requests are finished.

By combining an appropriate terminationGracePeriodSeconds with a signal-aware application, you bridge the gap between the speed of deployment and the reliability of user experience.

Key Takeaway: By setting terminationGracePeriodSeconds and coding your application to handle SIGTERM by finishing existing requests, you prevent dropped connections and ensure zero-downtime during rolling updates.