Integer vs Floating Point Representation
What you'll learn: How neural networks store numbers in memory and why choosing between FP32, FP16, and INT8 matters for speed, accuracy, and efficiency.
How Numbers Are Stored
At the hardware level, every number in your model—weights, activations, gradients—lives in memory as a sequence of bits. Floating point formats (like FP32 and FP16) store numbers with a sign, exponent, and fractional part, allowing wide ranges and decimal precision. Integers (like INT8) store whole numbers only, using far fewer bits.
The Three Main Formats
- FP32 (32-bit float): The standard. High precision, wide range (~10⁻³⁸ to 10³⁸), but uses 4 bytes per number.
- FP16 (16-bit float): Half the memory (2 bytes), faster on modern GPUs, but lower precision and smaller range (~10⁻⁸ to 65,000). Risk of overflow/underflow.
- INT8 (8-bit integer): Only 1 byte. Stores integers from -128 to 127 (or 0 to 255). Extremely fast and memory-efficient, but requires quantization—mapping floating point ranges to discrete integer bins.
The Tradeoffs
Think of it like choosing camera resolution: FP32 is 4K video (detailed, large files), FP16 is 1080p (good quality, manageable size), and INT8 is 480p (tiny, fast, but you lose detail).
| Format | Memory | Speed | Precision |
|---|---|---|---|
| FP32 | 4 bytes | 1× | High |
| FP16 | 2 bytes | ~2× | Medium |
| INT8 | 1 byte | ~4× | Low |
Quantization (converting FP32→INT8) trades precision for efficiency. A weight of 0.7341 might become 7 in INT8 after scaling, losing fine details but keeping the model's general behavior.
Key Takeaway: FP32 offers maximum accuracy but is memory-hungry and slow; FP16 balances speed and precision; INT8 maximizes efficiency at the cost of precision, requiring careful quantization to maintain model performance.