Course contentsShow
← Machine Learning and Deep Learning
Lesson 2976 of 3,53865. LLM Inference EnginesFree lesson

Attention Computation with Paged KV Cache

Modified attention kernel mechanics: gathering keys and values from non-contiguous blocks during forward pass.

Attention Computation with Paged KV Cache

What you'll learn: How attention kernels gather keys and values from non-contiguous memory blocks when using paged KV cache during inference.

The Challenge of Scattered Memory

In traditional attention, keys and values for a sequence are stored contiguously in memory—one continuous chunk per sequence. The attention kernel simply reads from predictable memory addresses. But with paged KV cache, your KV pairs are scattered across multiple non-contiguous blocks in physical memory, mapped through a page table.

How Modified Attention Works

The modified attention kernel must now perform a two-step lookup during the forward pass:

  1. Consult the page table: For each position in the sequence, determine which physical memory block holds that position's key and value
  2. Gather from blocks: Fetch the KV pairs from their scattered locations

Think of it like reading a book where pages are stored in different filing cabinets around a room. Instead of flipping sequentially, you need a map (page table) that tells you "page 5 is in cabinet C, drawer 2."

Kernel Modifications

The attention kernel receives:

  • Query tensor (as usual)
  • Block table (page table mapping logical positions → physical block IDs)
  • Block offset within each page
  • Physical memory pool containing all KV blocks

During computation, for each attention head position, the kernel calculates the block ID and offset, then gathers the appropriate K and V vectors before computing attention scores.

Performance Considerations

Modern implementations fuse the gather operation directly into the attention kernel to minimize memory traffic. The overhead is typically small (~5-10%) compared to the massive memory savings from better block utilization.

Key Takeaway: Paged attention kernels add an indirection layer—gathering KV pairs from scattered blocks via page table lookups—enabling memory-efficient serving at modest computational cost.