Model Compression for Serving
You'll learn how to shrink trained models without sacrificing much accuracy, making them faster and cheaper to deploy.
What Is Model Compression?
Model compression reduces a trained model's size and computational requirements so it runs faster during inference. Think of it like compressing a high-resolution photo for the web—you preserve the essential information while dramatically reducing file size and load time.
For classical ML models (tree ensembles, neural networks, SVMs), compression ensures your serving infrastructure handles more requests per second at lower cost.
Three Core Techniques
Pruning
Remove unnecessary weights or entire neurons/trees that contribute little to predictions. After pruning, you retrain briefly to recover any lost accuracy. It's like trimming dead branches from a tree—the structure becomes leaner without losing its core function.
Quantization
Reduce numerical precision of model parameters (e.g., float32 → int8). Instead of storing weights as 32-bit floating-point numbers, use 8-bit integers. This cuts memory usage by ~75% and speeds up CPU/GPU operations, though with slight accuracy trade-offs.
Knowledge Distillation
Train a smaller "student" model to mimic a larger "teacher" model's predictions. The student learns from the teacher's soft probability distributions, not just hard labels. You end up with a compact model that approximates the original's behavior.
Why It Matters for Serving
Compressed models:
- Load faster during container startup
- Use less memory, allowing more replicas per server
- Reduce inference latency per request
- Lower infrastructure costs significantly
These techniques work together—you might quantize a pruned model, then use it as the student in distillation.
Key Takeaway: Model compression through pruning, quantization, and knowledge distillation drastically reduces model size and latency, enabling cost-effective, high-performance serving at scale.