Course contentsShow
← AI Engineering
Lesson 1077 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Device Mapping Strategies

Automatic and manual device mapping to control which layers run on which GPUs for optimal performance.

Device Mapping Strategies

What you'll learn: How to control which model layers run on which GPUs for optimal performance and memory efficiency in multi-GPU setups.

The Core Challenge

When your LLM is too large for a single GPU, you need to split it across multiple devices. But not all layers are equal—some are memory-heavy, others compute-intensive. Device mapping is the strategy you use to decide which layers live on which GPU (or CPU) to balance memory usage and maximize throughput.

Automatic Device Mapping

Most frameworks offer automatic strategies that inspect your model architecture and available devices, then distribute layers intelligently:

  • "auto": Framework analyzes layer sizes and GPU memory, spreading layers evenly
  • "balanced": Aims for equal memory consumption across all GPUs
  • "sequential": Fills GPU 0 first, then GPU 1, etc.

Think of it like packing moving boxes: automatic strategies are like having a professional organizer decide what goes where based on weight and available truck space.

Manual Device Mapping

For fine-grained control, you specify exactly which layers go where using a device map dictionary:

device_map = {
    "model.embed_tokens": 0,
    "model.layers.0": 0,
    "model.layers.1": 0,
    "model.layers.2": 1,
    "model.layers.3": 1,
    "model.norm": 1,
    "lm_head": 1
}

This is useful when you know certain layers bottleneck performance or when one GPU has more free memory than others.

Why It Matters

Poor device mapping can cause GPUs to sit idle while waiting for data transfers between devices. Strategic mapping minimizes cross-device communication and keeps all hardware working efficiently.

Key Takeaway: Device mapping strategies—automatic or manual—let you distribute model layers across GPUs to balance memory constraints and optimize inference speed, turning multiple GPUs into a coordinated inference engine.