Course contentsShow
← Machine Learning and Deep Learning
Lesson 3355 of 3,53873. Privacy in Machine LearningFree lesson

Communication Efficiency in Federated Learning

Reducing communication costs through compression, quantization, and sparse updates.

Communication Efficiency in Federated Learning

What you'll learn: How to dramatically reduce network costs in federated learning by compressing, quantizing, and sparsifying model updates before transmission.

The Bottleneck Problem

In federated learning, hundreds or thousands of clients train local models and send updates back to a central server. The naive approach transmits full-precision gradient vectors or model weights—often millions or billions of parameters—every round. With limited bandwidth (especially on mobile networks), this becomes the primary bottleneck, far exceeding computation time.

Three Core Strategies

Compression

Apply lossy compression algorithms to model updates before sending. Think of it like converting a high-resolution photo to JPEG—you lose some fidelity but drastically reduce file size. Common techniques include top-k sparsification (sending only the largest k gradients) or random sparsification.

Quantization

Reduce numerical precision from 32-bit floats to 8-bit integers, 4-bit, or even 1-bit representations. Instead of sending the precise value -0.00347, you might send a quantized bin index. The server reconstructs approximate values. This can reduce communication by 4× to 32× with minimal accuracy loss.

Sparse Updates

Only transmit parameters that have changed significantly since the last round. Most gradient entries are near-zero—why send them? Sparse representations (storing only non-zero indices and values) can yield 100× compression in practice.

Practical Impact

These techniques often stack multiplicatively. Combining 10× top-k sparsification with 8× quantization yields 80× communication reduction, making federated learning viable on bandwidth-constrained devices. The tradeoff? Slightly slower convergence or minor accuracy degradation—but you enable training that otherwise couldn't happen at all.

Key Takeaway: Communication efficiency in federated learning uses compression, quantization, and sparse updates to reduce transmitted data by orders of magnitude, making distributed training practical under real-world bandwidth constraints.