Virtual Memory Concepts for LLM Serving
What You'll Learn
You'll learn how operating system virtual memory principles can solve the KV cache memory bottleneck by decoupling logical address space from physical GPU memory.
The Core Problem and Solution
Traditional LLM serving allocates contiguous physical GPU memory for each request's KV cache upfront. This is wasteful because:
- You must reserve maximum possible memory per request
- Fragmentation occurs as requests complete
- GPU memory is severely underutilized
Virtual memory for LLM serving borrows from OS memory management: separate what the model thinks it's accessing (logical addresses) from where data actually lives (physical memory).
How It Works
Think of it like apartment buildings vs. street addresses:
- Logical addresses: Each request gets a continuous "street address" for its KV cache (e.g., blocks 0-999)
- Physical memory: The actual GPU memory is divided into fixed-size pages (like apartments)
- Page table: A mapping that translates logical block IDs to physical memory locations
When the model needs KV cache block 42 for a request, the system looks up the page table and finds it's actually stored in physical page 156. The model doesn't know or care—it just uses logical block 42.
Benefits
This separation enables:
- On-demand allocation: Only allocate physical memory as the KV cache actually grows
- Memory sharing: Multiple requests can point to the same physical pages (useful for prefix sharing)
- Defragmentation: Move pages around without changing logical addresses
- Oversubscription: Logical space can exceed physical capacity (with eviction strategies)
Key Takeaway: Virtual memory for LLM serving decouples logical KV cache addresses from physical GPU memory locations, enabling dynamic allocation, sharing, and dramatically improved memory utilization—just like OS virtual memory does for RAM.