PagedAttention and vLLM
What You'll Learn
You'll learn how PagedAttention applies virtual memory techniques to manage KV cache efficiently, reducing fragmentation and enabling higher throughput in LLM serving.
The Memory Fragmentation Problem
Traditional KV cache allocation reserves contiguous memory blocks for each sequence's keys and values. This creates two major issues:
- Internal fragmentation: Pre-allocating for max sequence length wastes memory when sequences are shorter
- External fragmentation: Variable-length sequences leave gaps between allocations that can't be reused
The result? You can only serve a fraction of the requests your GPU memory could theoretically handle.
PagedAttention: Virtual Memory for Attention
PagedAttention, introduced by the vLLM team, borrows from operating system virtual memory:
- KV cache is divided into fixed-size blocks (or "pages")
- Each sequence's KV cache is stored in non-contiguous blocks
- A block table maps logical KV positions to physical memory blocks
Analogy: Instead of requiring a continuous bookshelf for each book series, you can store volumes on any available shelf space. A catalog tells you where each volume lives.
Key Benefits
- Near-zero waste: Blocks are only allocated as needed, and unused blocks are immediately available
- Memory sharing: Different sequences can point to the same physical blocks (perfect for prompt prefix caching)
- Higher throughput: 2-4x improvement in serving capacity compared to static allocation
vLLM Implementation
vLLM implements PagedAttention as its core memory management strategy, automatically handling block allocation, the block table, and attention computation across non-contiguous memory—transparent to the user.
Key Takeaway: PagedAttention eliminates KV cache fragmentation by using virtual memory techniques with fixed-size blocks, dramatically improving GPU memory utilization and serving throughput in systems like vLLM.