Course contentsShow
← AI Engineering
Lesson 1061 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Understanding Model Size and Memory Requirements

Calculate VRAM needs based on model parameters, precision, and inference settings for different deployment scenarios.

Understanding Model Size and Memory Requirements

You'll learn: How to calculate the video RAM (VRAM) your model will consume during inference based on its parameter count, precision format, and serving configuration.

Why Memory Math Matters

When deploying a self-hosted LLM, your first hard constraint is usually GPU memory. Loading a model that's "too big" will crash immediately or force inefficient CPU offloading. Understanding the relationship between model parameters and memory lets you pick the right hardware and configuration before you invest time or money.

The Core Formula

At its simplest, memory required for model weights equals:

Parameters × Bytes per Parameter = Base Memory

A 7B (7 billion parameter) model in FP16 precision (2 bytes per parameter) needs approximately:

  • 7,000,000,000 × 2 = 14 GB for weights alone

In FP32 (4 bytes): 28 GB
In INT8 (1 byte): 7 GB
In INT4 (0.5 bytes): 3.5 GB

Beyond the Weights

You must also budget for:

  • KV Cache: Grows with context length and batch size; can match or exceed model weight size
  • Activation Memory: Temporary tensors during forward passes
  • Framework Overhead: ~1-2 GB for libraries and buffers

A practical rule of thumb: multiply your base memory by 1.2-1.5× for single-user inference, or 2-3× for production batched serving.

Deployment Scenario Examples

  • 7B FP16 model, small context: 14 GB (weights) + 2 GB (cache) = ~16 GB → Fits on RTX 4090 (24 GB)
  • 13B INT4 quantized: 6.5 GB + 4 GB = ~10.5 GB → Fits on consumer GPUs
  • 70B FP16: 140 GB → Requires multi-GPU or aggressive quantization

Key Takeaway: Calculate VRAM needs by multiplying parameters by precision bytes, then adding 20-50% overhead for KV cache and activations—this determines your minimum GPU requirements.