Session Affinity and Load Balancing
What You'll Learn
You'll learn how to configure sticky sessions to route requests from the same user to the same server instance, ensuring consistent access to server-side session storage in distributed LLM applications.
The Core Idea
When you scale your LLM application horizontally (multiple server instances), a load balancer distributes incoming requests across these instances. Without configuration, User A's first request might go to Server 1, but their second request could land on Server 2—which doesn't have their conversation history if you're using server-side session storage.
Session affinity (also called "sticky sessions") solves this by making the load balancer remember which server handled a user's first request, then routing all subsequent requests from that user to the same server.
How It Works
Think of it like a restaurant host who remembers which waiter served you and always seats you at that waiter's tables during your visit. The load balancer uses identifiers—typically a session cookie or client IP address—to maintain this "stickiness."
Common configuration methods:
- Cookie-based affinity: Load balancer sets a cookie containing the target server ID
- IP-based affinity: Routes based on client IP address
- Header-based affinity: Uses custom headers to determine routing
The Trade-off
Session affinity enables server-side state but reduces load balancing flexibility. If Server 1 gets busy, new users assigned to it still go there. If Server 1 crashes, all its users lose their sessions unless you've implemented session replication or persistence.
Key Takeaway: Session affinity routes users to consistent server instances via load balancer configuration, enabling server-side session storage in distributed systems, but at the cost of some load balancing flexibility and potential session loss during server failures.