Communication Efficiency in Federated Learning
What you'll learn: How to dramatically reduce network costs in federated learning by compressing, quantizing, and sparsifying model updates before transmission.
The Bottleneck Problem
In federated learning, hundreds or thousands of clients train local models and send updates back to a central server. The naive approach transmits full-precision gradient vectors or model weights—often millions or billions of parameters—every round. With limited bandwidth (especially on mobile networks), this becomes the primary bottleneck, far exceeding computation time.
Three Core Strategies
Compression
Apply lossy compression algorithms to model updates before sending. Think of it like converting a high-resolution photo to JPEG—you lose some fidelity but drastically reduce file size. Common techniques include top-k sparsification (sending only the largest k gradients) or random sparsification.
Quantization
Reduce numerical precision from 32-bit floats to 8-bit integers, 4-bit, or even 1-bit representations. Instead of sending the precise value -0.00347, you might send a quantized bin index. The server reconstructs approximate values. This can reduce communication by 4× to 32× with minimal accuracy loss.
Sparse Updates
Only transmit parameters that have changed significantly since the last round. Most gradient entries are near-zero—why send them? Sparse representations (storing only non-zero indices and values) can yield 100× compression in practice.
Practical Impact
These techniques often stack multiplicatively. Combining 10× top-k sparsification with 8× quantization yields 80× communication reduction, making federated learning viable on bandwidth-constrained devices. The tradeoff? Slightly slower convergence or minor accuracy degradation—but you enable training that otherwise couldn't happen at all.
Key Takeaway: Communication efficiency in federated learning uses compression, quantization, and sparse updates to reduce transmitted data by orders of magnitude, making distributed training practical under real-world bandwidth constraints.