Course contentsShow
← Machine Learning and Deep Learning
Lesson 1691 of 3,53836. LLM Inference OptimizationFree lesson

Sparse vs Dense Models

Compare sparse MoE architectures to dense transformers in terms of parameters, FLOPs, memory, and effective capacity.

Sparse vs Dense Models

You'll learn how sparse MoE architectures compare to traditional dense transformers across key performance dimensions.

The Core Difference

A dense model activates every parameter for every input token—if you have a 7B parameter model, all 7 billion parameters compute something for each token. A sparse model like Mixture of Experts (MoE) contains many parameters but only activates a subset per token, routing through selected "experts" while leaving others dormant.

The Four-Way Comparison

Parameters vs Active Parameters

  • Dense: 7B parameters = 7B active parameters
  • Sparse MoE: 50B total parameters, but only 7B active per token (using 2 of 8 experts, for example)

FLOPs (Computation)

FLOPs scale with active parameters, not total. A sparse 50B MoE with 7B active parameters uses similar FLOPs to a 7B dense model during inference—much faster than a 50B dense model.

Memory Footprint

All parameters must live in GPU memory, whether active or not. Your 50B sparse model needs memory for all 50B parameters, even though only 7B compute at once. This is the cost of sparsity.

Effective Capacity

Here's the magic: sparse models can achieve performance closer to their total parameter count (50B) while running at the speed of their active parameter count (7B). More parameters = more knowledge stored, even if only accessed selectively.

The Trade-off Analogy

Think of a library: a dense model is one librarian who must read every book for every question. An MoE is a library with specialist librarians—you only consult the relevant experts, getting the benefit of a huge collection at the speed of asking one person.

Key Takeaway: Sparse MoE models trade increased memory requirements for better performance-per-FLOP, achieving larger effective capacity without proportional inference cost increases.