Handling 'Poison' Tenant Scenarios

In this lesson, you will learn how to isolate and quarantine messages associated with "poison" tenants to prevent a single corrupt data set from causing cascading failures across your background workers.

The Poison Tenant Problem

In multi-tenant architectures, a "poison" tenant is one whose specific data schema, volume, or edge-case constraints trigger unhandled exceptions in your processing logic. If you don't handle this, the message will fail, return to the queue, and be retried repeatedly—essentially creating a "poison pill" that blocks the worker from processing messages for healthy tenants.

Isolation through Quarantine

The goal is to move beyond simple dead-letter queues. Instead of dumping all failures into a single bucket, we implement a Tenant Isolation Strategy. When a consumer encounters a terminal failure, it identifies the TenantId associated with the event and routes that specific message to a dedicated "quarantine" area.

public async Task ProcessMessageAsync(EventEnvelope envelope)
{
    try 
    {
        await _handler.ExecuteAsync(envelope);
    }
    catch (Exception ex) when (IsPoisonous(ex))
    {
        // Isolate the tenant to prevent blocking others
        await _quarantineService.QuarantineTenantAsync(envelope.TenantId, envelope, ex.Message);
        
        // Acknowledge the message so it doesn't block the main queue
        await _queue.AckAsync(envelope); 
    }
}

Preventing Mass Failure

By quarantining the tenant, your background worker remains performant and free to process messages for the rest of your fleet. The quarantined events are persisted in a separate store, allowing you to run diagnostic scripts or manual data corrections for that specific tenant without impacting the overall system throughput. Once the underlying data issue is resolved, you can "thaw" the tenant and replay the quarantined events back into the primary stream.

This pattern ensures that your system maintains high availability even when one customer’s data creates a structural conflict with your current processing logic.

Key Takeaway: Rather than letting individual tenant failures block your message pipelines, implement an automated quarantine mechanism that extracts failing events by TenantId to maintain global worker throughput.