Attention Computation with Paged KV Cache
What you'll learn: How attention kernels gather keys and values from non-contiguous memory blocks when using paged KV cache during inference.
The Challenge of Scattered Memory
In traditional attention, keys and values for a sequence are stored contiguously in memory—one continuous chunk per sequence. The attention kernel simply reads from predictable memory addresses. But with paged KV cache, your KV pairs are scattered across multiple non-contiguous blocks in physical memory, mapped through a page table.
How Modified Attention Works
The modified attention kernel must now perform a two-step lookup during the forward pass:
- Consult the page table: For each position in the sequence, determine which physical memory block holds that position's key and value
- Gather from blocks: Fetch the KV pairs from their scattered locations
Think of it like reading a book where pages are stored in different filing cabinets around a room. Instead of flipping sequentially, you need a map (page table) that tells you "page 5 is in cabinet C, drawer 2."
Kernel Modifications
The attention kernel receives:
- Query tensor (as usual)
- Block table (page table mapping logical positions → physical block IDs)
- Block offset within each page
- Physical memory pool containing all KV blocks
During computation, for each attention head position, the kernel calculates the block ID and offset, then gathers the appropriate K and V vectors before computing attention scores.
Performance Considerations
Modern implementations fuse the gather operation directly into the attention kernel to minimize memory traffic. The overhead is typically small (~5-10%) compared to the massive memory savings from better block utilization.
Key Takeaway: Paged attention kernels add an indirection layer—gathering KV pairs from scattered blocks via page table lookups—enabling memory-efficient serving at modest computational cost.