Course contentsShow
← Machine Learning and Deep Learning
Lesson 2971 of 3,53865. LLM Inference EnginesFree lesson

Virtual Memory Concepts for LLM Serving

Applying operating system virtual memory principles to manage KV cache: separation of logical and physical memory.

Virtual Memory Concepts for LLM Serving

What You'll Learn

You'll learn how operating system virtual memory principles can solve the KV cache memory bottleneck by decoupling logical address space from physical GPU memory.

The Core Problem and Solution

Traditional LLM serving allocates contiguous physical GPU memory for each request's KV cache upfront. This is wasteful because:

  • You must reserve maximum possible memory per request
  • Fragmentation occurs as requests complete
  • GPU memory is severely underutilized

Virtual memory for LLM serving borrows from OS memory management: separate what the model thinks it's accessing (logical addresses) from where data actually lives (physical memory).

How It Works

Think of it like apartment buildings vs. street addresses:

  • Logical addresses: Each request gets a continuous "street address" for its KV cache (e.g., blocks 0-999)
  • Physical memory: The actual GPU memory is divided into fixed-size pages (like apartments)
  • Page table: A mapping that translates logical block IDs to physical memory locations

When the model needs KV cache block 42 for a request, the system looks up the page table and finds it's actually stored in physical page 156. The model doesn't know or care—it just uses logical block 42.

Benefits

This separation enables:

  • On-demand allocation: Only allocate physical memory as the KV cache actually grows
  • Memory sharing: Multiple requests can point to the same physical pages (useful for prefix sharing)
  • Defragmentation: Move pages around without changing logical addresses
  • Oversubscription: Logical space can exceed physical capacity (with eviction strategies)

Key Takeaway: Virtual memory for LLM serving decouples logical KV cache addresses from physical GPU memory locations, enabling dynamic allocation, sharing, and dramatically improved memory utilization—just like OS virtual memory does for RAM.