Understanding Model Size and Memory Requirements
You'll learn: How to calculate the video RAM (VRAM) your model will consume during inference based on its parameter count, precision format, and serving configuration.
Why Memory Math Matters
When deploying a self-hosted LLM, your first hard constraint is usually GPU memory. Loading a model that's "too big" will crash immediately or force inefficient CPU offloading. Understanding the relationship between model parameters and memory lets you pick the right hardware and configuration before you invest time or money.
The Core Formula
At its simplest, memory required for model weights equals:
Parameters × Bytes per Parameter = Base Memory
A 7B (7 billion parameter) model in FP16 precision (2 bytes per parameter) needs approximately:
- 7,000,000,000 × 2 = 14 GB for weights alone
In FP32 (4 bytes): 28 GB
In INT8 (1 byte): 7 GB
In INT4 (0.5 bytes): 3.5 GB
Beyond the Weights
You must also budget for:
- KV Cache: Grows with context length and batch size; can match or exceed model weight size
- Activation Memory: Temporary tensors during forward passes
- Framework Overhead: ~1-2 GB for libraries and buffers
A practical rule of thumb: multiply your base memory by 1.2-1.5× for single-user inference, or 2-3× for production batched serving.
Deployment Scenario Examples
- 7B FP16 model, small context: 14 GB (weights) + 2 GB (cache) = ~16 GB → Fits on RTX 4090 (24 GB)
- 13B INT4 quantized: 6.5 GB + 4 GB = ~10.5 GB → Fits on consumer GPUs
- 70B FP16: 140 GB → Requires multi-GPU or aggressive quantization
Key Takeaway: Calculate VRAM needs by multiplying parameters by precision bytes, then adding 20-50% overhead for KV cache and activations—this determines your minimum GPU requirements.