The Federated Averaging Algorithm
What you'll learn: How FedAvg orchestrates distributed training by coordinating local model updates across multiple clients and aggregating them into a single global model.
The Core Idea
Federated Averaging (FedAvg) is the foundational algorithm powering federated learning. Instead of sending raw data to a central server, FedAvg sends model updates. Here's how it works:
- Server initializes a global model and sends it to selected clients
- Clients train locally on their private data for several epochs using their own SGD
- Clients send back their updated model weights (not data!)
- Server averages all client updates, weighted by dataset size
- Repeat until convergence
Why Local Training Rounds Matter
Each client performs multiple gradient steps before communicating. This reduces the number of communication rounds needed—crucial because network communication is often the bottleneck in federated settings. Think of it like a study group where everyone works independently on practice problems for a week, then meets briefly to share insights, rather than texting about every single problem.
The Aggregation Step
The server combines client models using a weighted average:
w_global = Σ(n_k / n_total) × w_k
where n_k is client k's dataset size and w_k is their model weights. Clients with more data contribute proportionally more to the global model, ensuring fairness and statistical efficiency.
Why FedAvg Succeeds
FedAvg balances privacy (data never leaves devices), efficiency (fewer communication rounds), and performance (converges to models competitive with centralized training under the right conditions).
Key Takeaway: FedAvg coordinates federated learning by having clients perform multiple local training epochs independently, then aggregating their model updates through weighted averaging based on dataset sizes—minimizing communication while preserving privacy.