{ "id": "ep3ajjeqqm", "slug": "federated-averaging", "title": "Federated averaging", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Non-geographic · 2001 to 2020: Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods" } ] } ], "excerpt": "Federated averaging (FedAvg) is a distributed machine learning algorithm, reported by McMahan and colleagues in 2016, in which a server averages locally trained models from clients holding private data.", "snippet": "Federated averaging (FedAvg) is a distributed machine learning algorithm, reported by McMahan and colleagues in 2016, in which a server averages locally trained models from clients holding private data.", "node": "technology.computing.ai.ml_methods", "markdown": "# Federated averaging\n\nFederated averaging (FedAvg) is a distributed machine learning algorithm in which a central server combines locally trained model weights from many clients that hold private data, iteratively training a shared global model without exchanging raw data. It combines local stochastic gradient descent (SGD) on each client with server-side model averaging, and was reported as reducing required communication rounds by 10 to 100 times compared with a naively federated version of SGD, while remaining robust to unbalanced and non-IID (non-independent and identically distributed) data.[1](https://doi.org/10.48550/arxiv.1602.05629) It has been described as the first and perhaps most widely used federated learning algorithm,[2](https://arxiv.org/pdf/1907.02189) in a setting defined by data spread across massive numbers of unreliable devices.[3](https://arxiv.org/pdf/1912.04977)\n\n| Key fact | Detail |\n|---|---|\n| What it produces | A single shared global model, trained over communication rounds in which clients train locally and the server averages their weights[1](https://doi.org/10.48550/arxiv.1602.05629) |\n| Communication savings | 10–100× fewer rounds than naively federated SGD[1](https://doi.org/10.48550/arxiv.1602.05629) |\n| Control parameters | C (fraction of clients computing per round), E (local passes per round), B (local minibatch size)[1](https://doi.org/10.48550/arxiv.1602.05629) |\n| Aggregation rule | Size-weighted average of client models, weighted by local dataset size[4](https://scalablebook.apartsin.com/part-3-distributed-ml/module-14-federated-decentralized-learning/section-14.3.html) |\n| Main failure mode | Client drift under non-IID data, which grows with the number of local epochs E |\n| Privacy | No raw data leaves devices, but gradient updates can leak training data; differential privacy adds a utility cost[5](https://arxiv.org/html/2206.12395v3) |\n| Practical deployment | Tested in Google Gboard on Android for query suggestion models, announced April 6, 2017[6](https://research.google/blog/federated-learning-collaborative-machine-learning-without-centralized-training-data/) |\n\n## How it works\n\nThe baseline is FedSGD: each client computes a gradient on its local data, and the server takes the step \\( w_{t+1} \\leftarrow w_t - \\eta \\sum_{k} (n_{k}/n) \\, g_{k} \\), a gradient average in which each client k is weighted by its local dataset size \\( n_{k} \\) out of the total n.[1](https://doi.org/10.48550/arxiv.1602.05629) FedAvg generalizes this by letting each client run several local updates before communicating, then averaging the resulting models rather than the gradients. The server computes the size-weighted average \\( w^{(t+1)} = \\sum_{k \\in S_{t}} (n_{k}/N) \\, w_{k}^{(t+1)} \\), where \\( N = \\sum_{k \\in S_{t}} n_{k} \\) is the total number of examples across the participating clients, so a client with twice the data pulls twice as hard on the global model. With \\( E = 1 \\) and one full-batch local step per client, this weighted model average equals exactly one gradient step on the global objective, making FedAvg with a single local step equivalent to centralized SGD.\n\nFor non-IID strongly convex and smooth problems, FedAvg has a proven \\( \\mathcal{O}(1/T) \\) convergence rate, exposing a trade-off between communication efficiency and convergence rate.[2](https://arxiv.org/pdf/1907.02189) Hyperparameters enter the guarantees directly: with a fixed learning rate \\( \\eta \\) and \\( E > 1 \\), FedAvg converges to a solution at least \\( \\Omega(\\eta(E-1)) \\) away from the optimum, so learning-rate decay is necessary; and E must not exceed \\( \\Omega(\\sqrt{T}) \\), otherwise convergence is not guaranteed.[2](https://arxiv.org/pdf/1907.02189) Under non-IID data the rate depends only weakly on the number of participating devices K, so FedAvg cannot achieve linear speedup, but low participation ratios can be used without slowing learning.[2](https://arxiv.org/pdf/1907.02189) A caveat from non-parametric analysis: FedAvg with aggregation period \\( s > 1 \\) and FedProx fail to reach the stationary point of the global objective even for homogeneous linear regression, yet both converge to nearly the same estimation error as one-step-per-round FedAvg, with convergence time shrinking roughly by a factor of s.[7](https://jmlr.org/papers/volume24/22-0153/22-0153.pdf)\n\n## How it is done\n\nA practitioner runs a repeating loop described in the standard federated learning template: client selection, broadcast of the current model, client computation, aggregation, and model update; stragglers may be dropped at aggregation once enough devices have reported.[3](https://arxiv.org/pdf/1912.04977) Three parameters control the amount of client computation: C, the fraction of clients that perform computation on each round; E, the number of training passes each client makes over its local dataset; and B, the local minibatch size. A client with \\( n_{k} \\) local examples performs \\( u_{k} = E \\cdot n_{k}/B \\) local updates per round.[1](https://doi.org/10.48550/arxiv.1602.05629) In the TensorFlow Federated reference implementation, client deltas are aggregated with a default mean factory, client weighting defaults to the number of examples (with a uniform option), and the default server optimizer is SGD with learning rate 1.0, which simply adds the averaged delta to the server model and recovers the original FedAvg.[8](https://github.com/tensorflow/federated/blob/610843c724740e1b041837cc93501b609fb05d8f/tensorflow_federated/python/learning/federated_averaging.py) In Google's on-device deployment, training runs a miniature [TensorFlow](https://www.edgepedia.org/tensorflow) scheduled only when the device is idle, plugged in, and on a free wireless connection.[6](https://research.google/blog/federated-learning-collaborative-machine-learning-without-centralized-training-data/)\n\n## Origin\n\nFedAvg and the term federated learning were reported by McMahan and colleagues in a preprint posted in February 2016, later published at AISTATS 2017 (PMLR 54:1273-1282).[1](https://doi.org/10.48550/arxiv.1602.05629) The motivating application was mobile keyboards: Google announced on April 6, 2017 that it was testing federated learning in [Gboard](https://www.edgepedia.org/gboard) on Android for query suggestion models.[6](https://research.google/blog/federated-learning-collaborative-machine-learning-without-centralized-training-data/) The original paper credits earlier work it builds on: Shokri & Shmatikov (2015) as the most relevant prior work, training deep networks with shared subsets of parameters and global differential privacy; Zinkevich et al. (2011), who studied a very similar averaging algorithm in the convex, balanced, IID setting; distributed averaging by McDonald et al. (perceptrons) and Povey et al. (speech DNNs); asynchronous soft averaging by Zhang et al.; and one-shot averaging in the convex IID case.[1](https://doi.org/10.48550/arxiv.1602.05629) Later theory analyzed the same algorithm under the names Local SGD or parallel SGD, before the federated learning framing.[9](https://proceedings.mlr.press/v151/glasgow22a/glasgow22a.pdf) Konečný and colleagues proposed structured and sketched updates to cut communication, using FedAvg as the base algorithm in all experiments.[10](https://doi.org/10.48550/arxiv.1610.05492)\n\n## Variants\n\n**FedProx** adds a proximal term \\( (\\mu/2)\\lVert w - w_{t}\\rVert^{2} \\) to each local objective and tolerates variable amounts of local work (γ-inexact solutions) instead of a uniform E; in simulations with 90% stragglers it improved absolute test accuracy over FedAvg by 22% on average.[11](https://proceedings.mlsys.org/paper_files/paper/2020/file/1f5fe83998a09396ebe6477d9475ba0c-Paper.pdf) **SCAFFOLD**, reported by Karimireddy and colleagues, corrects client drift with control variates: a server control variate c and per-client \\( c_{i} \\), where the difference \\( c - c_{i} \\) estimates the client drift and corrects the local update; it requires significantly fewer communication rounds and is not affected by data heterogeneity or client sampling.[12](https://proceedings.mlr.press/v119/karimireddy20a.html) **FedNova**, reported by Wang and colleagues in 2020, is a normalized averaging method that eliminates objective inconsistency caused by heterogeneous numbers of local updates; on non-IID CIFAR-10 over 100 rounds, simply changing the aggregation weights yielded a 6-9% test-accuracy improvement when the client optimizer is SGD or SGD with momentum.[13](https://doi.org/10.48550/arxiv.2007.07481) Server-side adaptive optimizers have also been benchmarked: in the FedScale evaluation, FedYogi performed best on OpenImage but was inferior to FedAvg on Google Speech, so the preferred optimizer is task-dependent.[14](https://symbioticlab.org/publications/files/fedscale:icml22/fedscale-icml22.pdf) An extension adding a server learning rate, distinct from classic FedAvg and reducing to it when the server learning rate equals 1, is studied in the convergence literature.[9](https://proceedings.mlr.press/v151/glasgow22a/glasgow22a.pdf) On the communication side, structured and sketched updates (low-rank and random-mask constraints; subsampling, quantization, sketching) reduce total communicated data by two orders of magnitude with slight convergence degradation; sketching that keeps 6.25% of elements with 2-bit quantization saves a factor of 256 in bits.[10](https://doi.org/10.48550/arxiv.1610.05492) More recently, practice has shifted toward federated fine-tuning of large language models, where a 2025 survey identifies four challenges: communication overhead (billions of parameters), data heterogeneity (weight divergence and slower convergence), a memory wall on edge clients, and computation overhead.[15](https://arxiv.org/pdf/2503.12016) Named lines include FedIT, which integrates LoRA into classic FedAvg for instruction tuning, and FedSA-LoRA, which uploads only the A matrices to the server because they primarily encode general knowledge while the B matrices capture client-specific features.[15](https://arxiv.org/pdf/2503.12016)\n\n## Applications\n\nOn IID-partitioned MNIST, more local computation cut the rounds needed to reach target accuracy by 35× for a CNN and 46× for a two-layer network; on pathological non-IID partitions the speedups were 2.8-3.7×.[1](https://doi.org/10.48550/arxiv.1602.05629) On the MNIST CNN, FedSGD (\\( B = \\infty, E = 1 \\)) reached 99.22% accuracy after 1200 rounds, while FedAvg (\\( B = 10, E = 20 \\)) reached 99.44% after 300 rounds; the authors conjecture that model averaging also gives a regularization benefit similar to dropout.[1](https://doi.org/10.48550/arxiv.1602.05629) On CIFAR-10, standard SGD reached 86% test accuracy after 197,500 minibatch updates (each requiring a communication round in the federated setting), while FedAvg reached 85% after 2,000 communication rounds; on a large-scale LSTM task, FedSGD needed 820 rounds to reach 10.5% accuracy versus 35 rounds for FedAvg, 23× fewer.[1](https://doi.org/10.48550/arxiv.1602.05629) Against centralized training, benchmarks show the two are practically equivalent on IID data, while the centralized approach outperforms FedAvg with non-IID data.[16](https://dl.acm.org/doi/10.1145/3286490.3286559) Realistic FedScale benchmarks quantify the heterogeneity gap: FEMNIST ResNet-18 86.40% (IID) versus 78.50% (non-IID), OpenImage MobileNet-V2 80.83% versus 70.09%, Google Speech ResNet-34 72.58% versus 63.37%, and Reddit language modeling 73.5 versus 77.3 perplexity.[14](https://symbioticlab.org/publications/files/fedscale:icml22/fedscale-icml22.pdf) In federated fine-tuning of large language models, locally fine-tuned models degrade by up to 7% on the MMLU benchmark compared with federated fine-tuning.[15](https://arxiv.org/pdf/2503.12016)\n\n## Limitations and alternatives\n\nClient drift is zero when all clients hold identical data and grows with both the number of local epochs E and the dissimilarity between client datasets; every FedAvg variant attempts to enlarge the usable range of E by shrinking that gap. At the extremes, averaging two MNIST models trained from different initial conditions produces bad behavior, showing that parameter-space averaging can fail for non-convex objectives, and for very large numbers of local epochs FedAvg can plateau or diverge.[1](https://doi.org/10.48550/arxiv.1602.05629) Not sending raw data does not make updates private: gradient-leakage attacks let an honest-but-curious server reconstruct clients' private data from gradient updates alone, and reconstruction has been extended to FedAvg, where the server observes only aggregates of client updates after multiple local epochs; reconstruction is harder but still possible, using a gradient-similarity loss that simulates the hidden local steps plus an epoch order-invariant prior.[5](https://arxiv.org/html/2206.12395v3) [Differential privacy](https://www.edgepedia.org/differential-privacy) provides rigorous worst-case guarantees but at a utility cost: at privacy target \\( \\sigma = 0.01 \\), DP-SGD degraded final accuracy by 12.8% with \\( N = 30 \\) participants but only 4.6% with \\( N = 100 \\).[14](https://symbioticlab.org/publications/files/fedscale:icml22/fedscale-icml22.pdf) Alternatives include split learning, which splits model execution per layer between clients and server and sends only \"smashed data\" at a cut layer, and fully decentralized learning, which replaces server communication with peer-to-peer averaging between neighboring clients; personalized per-client models can turn the non-IID problem from a bug into a feature.[3](https://arxiv.org/pdf/1912.04977)\n\n## References\n\n1. [McMahan, H. Brendan and colleagues (2016). Communication-Efficient Learning of Deep Networks from Decentralized Data. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1602.05629)\n2. [On the Convergence of FedAvg on Non-IID Data](https://arxiv.org/pdf/1907.02189)\n3. [Advances and Open Problems in Federated Learning (Kairouz et al. survey)](https://arxiv.org/pdf/1912.04977)\n4. [Section 14.3: FedAvg and Its Variants | Scaling Out AI](https://scalablebook.apartsin.com/part-3-distributed-ml/module-14-federated-decentralized-learning/section-14.3.html)\n5. [Data Leakage in Federated Averaging](https://arxiv.org/html/2206.12395v3)\n6. [Federated Learning: Collaborative Machine Learning without Centralized Training Data (Google AI Blog, April 6, 2017)](https://research.google/blog/federated-learning-collaborative-machine-learning-without-centralized-training-data/)\n7. [A Non-parametric View of FedAvg and FedProx: Beyond Stationary Points](https://jmlr.org/papers/volume24/22-0153/22-0153.pdf)\n8. [tensorflow_federated federated_averaging.py (reference implementation)](https://github.com/tensorflow/federated/blob/610843c724740e1b041837cc93501b609fb05d8f/tensorflow_federated/python/learning/federated_averaging.py)\n9. [Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective](https://proceedings.mlr.press/v151/glasgow22a/glasgow22a.pdf)\n10. [Konečný, Jakub and colleagues (2016). Federated Learning: Strategies for Improving Communication Efficiency. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1610.05492)\n11. [Federated Optimization in Heterogeneous Networks (FedProx)](https://proceedings.mlsys.org/paper_files/paper/2020/file/1f5fe83998a09396ebe6477d9475ba0c-Paper.pdf)\n12. [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://proceedings.mlr.press/v119/karimireddy20a.html)\n13. [Wang, Jianyu and colleagues (2020). Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2007.07481)\n14. [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://symbioticlab.org/publications/files/fedscale:icml22/fedscale-icml22.pdf)\n15. [Survey of federated fine-tuning of large language models (FedLLM)](https://arxiv.org/pdf/2503.12016)\n16. [A Performance Evaluation of Federated Learning Algorithms](https://dl.acm.org/doi/10.1145/3286490.3286559)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/federated-averaging", "markdown_url": "https://www.edgepedia.org/federated-averaging.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Federated averaging\", Edgepedia (EdgeChat), https://www.edgepedia.org/federated-averaging. Edgepedia Community License 1.0.", "credit_md": "\"[Federated averaging](https://www.edgepedia.org/federated-averaging)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/federated-averaging](https://www.edgepedia.org/federated-averaging). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Federated averaging\", Edgepedia (EdgeChat), https://www.edgepedia.org/federated-averaging. Edgepedia Community License 1.0.", "speakable": "Federated averaging is a distributed machine learning algorithm, reported by McMahan and colleagues in 2016, in which a server averages locally trained models from clients holding private data." }