Graceful Server Shutdown Synchronization

In this lesson, you will learn how to orchestrate a graceful shutdown of your SignalR servers to prevent abrupt connection drops and ensure that your clients migrate to healthy instances without losing state.

The Challenge of Abrupt Termination

When a web server receives a SIGTERM signal (common during a rolling deployment or autoscaling event), it typically stops accepting new requests immediately. In real-time services, cutting a WebSocket connection mid-session causes a "thundering herd" effect: thousands of clients simultaneously attempt to reconnect, potentially overwhelming your load balancer or your remaining active nodes.

Orchestrating the "Drain" Phase

To achieve a graceful shutdown, we must introduce a Drain Phase. The goal is to notify clients that the server is entering maintenance mode, allow them to rebalance, and only then terminate the process.

1. Signal the Clients: Before the application process stops, send a broadcast to all connected clients using Clients.All.SendAsync("ServerClosing").
2. Delay Execution: Introduce a short delay (e.g., 5-10 seconds) after the broadcast. This gives clients enough time to receive the message and trigger their own connection migration logic.
3. Prevent New Joins: Use a custom IConnectionHandler or a state flag to stop accepting new connection handshakes while the drain is in progress.

Here is a conceptual implementation using the IHostApplicationLifetime hook:

public class GracefulShutdownService : IHostedService
{
    private readonly IHubContext<NotificationHub> _hubContext;
    private readonly IHostApplicationLifetime _lifetime;

    public GracefulShutdownService(IHubContext<NotificationHub> hubContext, IHostApplicationLifetime lifetime)
    {
        _hubContext = hubContext;
        _lifetime = lifetime;
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _lifetime.ApplicationStopping.Register(async () =>
        {
            // Notify clients to reconnect to a different node
            await _hubContext.Clients.All.SendAsync("OnServerShutdown", CancellationToken.None);
            
            // Allow a grace period for clients to migrate
            await Task.Delay(5000);
        });
        return Task.CompletedTask;
    }
    
    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

By decoupling the shutdown signal from the process termination, you shift from a destructive "hard cut" to a synchronized migration. This keeps your system stable, as clients spread their reconnection attempts over the grace period rather than slamming the infrastructure at the exact moment of deployment.

Key Takeaway: Graceful shutdown in SignalR relies on notifying clients to migrate before the process exits, effectively turning a service interruption into an orderly rebalancing event.