{ "id": "eptpdy1v9s", "slug": "model-agnostic-meta-learning", "title": "Model-agnostic meta-learning", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" }, { "id": "technology.computing.ai.ensembles_boosting", "label": "Ensemble, boosting, and transfer methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ensembles_boosting" }, { "id": "technology.computing.ai.ensembles_boosting.transfer_learning", "label": "Transfer learning and domain adaptation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ensembles_boosting.transfer_learning" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.ensembles_boosting", "label": "Non-geographic · 2001 to 2020: Ensemble, boosting, and transfer methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ensembles_boosting", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ensembles_boosting", "label": "Ensemble, boosting, and transfer methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ensembles_boosting" } ] } ], "excerpt": "Model-agnostic meta-learning (MAML) is a meta-learning algorithm, introduced by Chelsea Finn and colleagues in 2017, that trains initial parameters for fast adaptation to new tasks.", "snippet": "Model-agnostic meta-learning (MAML) is a meta-learning algorithm, introduced by Chelsea Finn and colleagues in 2017, that trains initial parameters for fast adaptation to new tasks.", "node": "technology.computing.ai.ensembles_boosting.transfer_learning", "markdown": "# Model-agnostic meta-learning\n\nModel-agnostic meta-learning (MAML) is a meta-learning algorithm that trains a model's initial parameters so that a few gradient descent steps on a small amount of new-task data adapt the model quickly to that task. In effect, it trains the model to be easy to fine-tune, and it applies to any model trained by gradient descent, including fully connected, convolutional, and recurrent networks under supervised or reinforcement learning losses.[1](https://doi.org/10.48550/arxiv.1703.03400)\n\n| Key fact | Detail |\n|---|---|\n| Output of training | A set of initial parameters θ from which one or a few gradient steps reach good task-specific performance[1](https://doi.org/10.48550/arxiv.1703.03400) |\n| Introduced by | Chelsea Finn, Pieter Abbeel, and Sergey Levine, ICML 2017 (volume 70, pages 1126–1135)[2](https://dl.acm.org/doi/10.5555/3305381.3305498) |\n| Core update | Inner loop θ′ᵢ = θ − α∇θL_Tᵢ(f_θ); outer loop θ ← θ − β∇θ Σ L_Tᵢ(f_θ′ᵢ)[1](https://doi.org/10.48550/arxiv.1703.03400) |\n| MiniImageNet 5-way | 48.70 ± 1.84% (1-shot) and 63.11 ± 0.92% (5-shot)[1](https://doi.org/10.48550/arxiv.1703.03400) |\n| First-order variant | FOMAML removes Hessian-vector products, about 33% faster, with nearly the same accuracy[1](https://doi.org/10.48550/arxiv.1703.03400) |\n| Memory limit | MAML's memory grows linearly with inner gradient steps, filling a 12 GB GPU in about 16 steps on 20-way 5-shot Omniglot[3](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf) |\n| Standing | No longer state of the art on few-shot benchmarks, surpassed by LEO and MetaOptNet[4](https://link.springer.com/article/10.1007/s10994-022-06210-y) |\n\n## How it works\n\nMAML consists of two nested stages. The inner loop takes a few gradient steps on a sampled task's small dataset, producing adapted parameters; the outer loop updates the initialization so that performance after adaptation, measured on each task, is high. In the one-step form, the meta-objective is \\( \\min_{\\theta} \\; F(\\theta) := \\mathbb{E}_{i}[ f_{i}(\\theta - \\alpha \\nabla f_{i}(\\theta)) ] \\), seeking an initialization that performs well after one gradient step on a new task.[5](https://proceedings.mlr.press/v108/fallah20a/fallah20a.pdf)\n\nBecause the outer objective depends on the inner gradient step, the meta-gradient is a gradient through a gradient. Computing it requires an additional backward pass through the model to obtain Hessian-vector products, which standard deep learning libraries such as [TensorFlow](https://www.edgepedia.org/tensorflow) support.[1](https://doi.org/10.48550/arxiv.1703.03400) A Taylor-series analysis of the Reptile paper explains why such initializations help: SGD itself produces a second-order term that adjusts the initial weights to maximize the dot product between gradients of different minibatches of the same task, encouraging gradients to generalize across the task's data.[6](https://arxiv.org/pdf/1803.02999)\n\n## How it is done\n\nEach meta-iteration samples a batch of tasks. For each task, the inner stage runs a few steps of (stochastic) gradient descent from the current initialization θ with a pre-defined step size α; the outer stage then updates θ with meta-learning rate β over all sampled tasks, using the adapted parameters.[1](https://doi.org/10.48550/arxiv.1703.03400) • [7](https://par.nsf.gov/servlets/purl/10339904) The two rates have distinct roles: α controls task-specific adaptation, β controls meta-learning across tasks.[8](https://www.arxiv.org/pdf/2602.19837)\n\nReported settings show the ranges involved. Omniglot 5-way models used one gradient step with \\( \\alpha = 0.4 \\) and a meta batch size of 32 tasks; MiniImageNet models used 5 training steps of \\( \\alpha = 0.01 \\), 10 evaluation steps, and 60000 iterations on a single NVIDIA Pascal Titan X GPU.[1](https://doi.org/10.48550/arxiv.1703.03400) Convergence theory adds a constraint: for multi-step MAML with N inner steps, the inner step size must be chosen inversely proportional to N (on the order of \\( 1/(NL) \\)) for guaranteed convergence in the general nonconvex setting.[7](https://par.nsf.gov/servlets/purl/10339904) [Performance](https://www.edgepedia.org/performance) and stability depend heavily on α and β, which in practice typically require exhaustive grid search.[9](https://arxiv.org/abs/1905.07435)\n\n## Origin\n\nMAML was introduced by [Chelsea Finn](https://www.edgepedia.org/chelsea-finn), Pieter Abbeel, and [Sergey Levine](https://www.edgepedia.org/sergey-levine) in \"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks\", posted to arXiv in March 2017 and published at ICML 2017 in volume 70 of the proceedings, pages 1126–1135.[1](https://doi.org/10.48550/arxiv.1703.03400) • [2](https://dl.acm.org/doi/10.5555/3305381.3305498) The paper builds on earlier work it cites, including \"Learning to learn by gradient descent by gradient descent\" by Marcin Andrychowicz and colleagues (2016), and matching networks by [Oriol Vinyals](https://www.edgepedia.org/oriol-vinyals) and colleagues (2016).[10](https://doi.org/10.48550/arxiv.1606.04474) • [11](https://doi.org/10.48550/arxiv.1606.04080)\n\n## Variants\n\n**First-order approximations** drop the second derivatives. FOMAML, proposed alongside MAML by ignoring the second derivative terms, removes the Hessian-vector products and gave roughly 33% speed-up in network computation.[1](https://doi.org/10.48550/arxiv.1703.03400) • [6](https://arxiv.org/pdf/1803.02999) It assumes the second-order gradient factor is the identity matrix, which increases efficiency in time and memory while achieving roughly the same performance as second-order MAML, though it does not compute an accurate meta-gradient because it ignores the Jacobian.[3](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf) • [12](https://link.springer.com/article/10.1007/s10994-023-06387-w) Reptile repeatedly samples a task, trains on it, and moves the initialization toward the trained weights via \\( \\phi \\leftarrow \\phi + \\epsilon (W - \\phi) \\), with no differentiation through the optimization process.[6](https://arxiv.org/pdf/1803.02999) On Omniglot and Mini-ImageNet, MAML, FOMAML, and Reptile performed very similarly, with Reptile slightly better on Mini-ImageNet (49.97 ± 0.32% versus 48.70 ± 1.84% in 5-way 1-shot) and slightly worse on Omniglot.[6](https://arxiv.org/pdf/1803.02999) • [3](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf)\n\n**Hessian-free and implicit methods** keep accuracy while avoiding second-order computation. HF-MAML recovers MAML's convergence bounds without second-order information at O(d) per-iteration cost, whereas FO-MAML has the same O(d) cost but cannot reach arbitrarily small accuracy.[5](https://proceedings.mlr.press/v108/fallah20a/fallah20a.pdf) iMAML computes meta-gradients by implicit differentiation, depending only on the inner-loop solution rather than the optimization path, with a memory footprint no larger than a single inner-loop gradient.[3](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf) ES-MAML offers another Hessian-free approach.[13](https://doi.org/10.48550/arxiv.1910.01215)\n\n**Probabilistic and adaptive variants** reinterpret or extend the objective. LLAMA reformulates MAML as probabilistic inference in a hierarchical Bayesian model, interpreting the initialization as parameters of a prior over task-specific parameters and adaptation as MAP inference, and replaces the point estimate with a Laplace approximation.[14](https://cocosci.princeton.edu/papers/gradient-based_meta-learning.pdf) BMAML combines gradient-based meta-learning with [Stein variational gradient descent](https://www.edgepedia.org/stein-variational-gradient-descent) over multiple particles, reducing to MAML when a single particle is used; its chaser loss is a first-order meta-update that prevents meta-level overfitting.[15](https://proceedings.neurips.cc/paper/2018/file/e1021d43911ca2c1845910d84f40aeae-Paper.pdf) PLATIPUS (Finn, Xu, and Levine, 2018) adds a probabilistic treatment of MAML itself.[16](https://doi.org/10.48550/arxiv.1806.02817) Alpha MAML adapts α and β online by hypergradient descent with virtually no extra gradient computations; in cases where standard MAML failed to reach a loss threshold within 400 iterations, it often reached the threshold in fewer than 50. Sharp-MAML (Momin Abbas and colleagues, 2022) adds sharpness awareness to the meta-objective.[17](https://doi.org/10.48550/arxiv.2206.03996) Multi-step variants include ANIL and BOIL.[7](https://par.nsf.gov/servlets/purl/10339904)\n\n## Applications\n\nThe original paper demonstrated three settings. On few-shot classification, MAML reached 48.70 ± 1.84% (1-shot) and 63.11 ± 0.92% (5-shot) on 5-way MiniImageNet, against 43.56 ± 0.84% and 55.31 ± 0.73% for matching networks and 43.44 ± 0.77% and 60.60 ± 0.71% for the meta-learner LSTM.[1](https://doi.org/10.48550/arxiv.1703.03400) On 5-shot sinusoid regression, MAML achieved a mean squared error of 0.67 after one gradient step versus 2.41 for pretraining on all tasks.[1](https://doi.org/10.48550/arxiv.1703.03400) In reinforcement learning, a policy with two hidden layers of size 100 used [REINFORCE](https://www.edgepedia.org/reinforce) for the adaptation gradients and TRPO as the meta-optimizer, adapting to new goal velocities and directions in two or three gradient steps, much faster than pretraining or random initialization.[1](https://doi.org/10.48550/arxiv.1703.03400)\n\n## Limitations and alternatives\n\n**Cost and memory.** The exact meta-gradient requires Hessian-vector products, which makes implementation costly.[5](https://proceedings.mlr.press/v108/fallah20a/fallah20a.pdf) Memory grows linearly with the number of inner gradient steps, reaching the capacity of a 12 GB GPU in approximately 16 steps on 20-way 5-shot Omniglot.[3](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf) First-order approximations trade this cost away, but FO-MAML cannot achieve arbitrarily small optimization error, while HF-MAML and iMAML recover accuracy without second-order information.[5](https://proceedings.mlr.press/v108/fallah20a/fallah20a.pdf) • [3](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf)\n\n**Meta-overfitting and RL performance.** In reinforcement learning, MAML can lag memory-based meta-learners: on MuJoCo's HalfCheetahVel task it achieves an average reward of −150 while VariBAD, TrMRL, and RL² achieve around −25 from the first episode, motivating extensions such as Robust MAML and XB-MAML.[8](https://www.arxiv.org/pdf/2602.19837) BMAML's chaser loss was proposed specifically to prevent meta-level overfitting.[15](https://proceedings.neurips.cc/paper/2018/file/e1021d43911ca2c1845910d84f40aeae-Paper.pdf)\n\n**Comparison with fine-tuning and metric methods.** When evaluation tasks come from a different data distribution than training, simply fine-tuning a pre-trained network may be more effective than MAML; MAML and Reptile tend to outperform fine-tuning when the distributions are similar, because they specialize in fast adaptation in low-data regimes. The pre-trained features from the fine-tuning baseline are more diverse and discriminative than those learned by MAML and Reptile, explaining the failure to generalize out of distribution.[12](https://link.springer.com/article/10.1007/s10994-023-06387-w) On few-shot classification benchmarks, MAML no longer yields state-of-the-art performance, having been surpassed by latent embedding optimization (LEO), which optimizes initial weights in a lower-dimensional latent space, and MetaOptNet, which stacks a convex model on a meta-learned feature extractor.[4](https://link.springer.com/article/10.1007/s10994-022-06210-y) MetaICL (Sewon Min and colleagues, 2021) applies meta-learning to language models through in-context learning without MAML's two-step optimization.[18](https://doi.org/10.48550/arxiv.2110.15943) How MAML-style optimization compares in detail with in-context learning in large foundation models after 2023 has not been settled in published comparisons.\n\n## References\n\n1. [Finn, Chelsea, Abbeel, Pieter, Levine, Sergey (2017). Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1703.03400)\n2. [Model-agnostic meta-learning for fast adaptation of deep networks | Proceedings of the 34th ICML (ACM DL record)](https://dl.acm.org/doi/10.5555/3305381.3305498)\n3. [Meta-Learning with Implicit Gradients (iMAML, NeurIPS 2019)](https://proceedings.neurips.cc/paper_files/paper/2019/file/072b030ba126b2f4b2374f342be9ed44-Paper.pdf)\n4. [Stateless neural meta-learning using second-order gradients (TURTLE; Machine Learning journal, 2022)](https://link.springer.com/article/10.1007/s10994-022-06210-y)\n5. [On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms (Fallah, Mokhtari, Ozdaglar, AISTATS 2020)](https://proceedings.mlr.press/v108/fallah20a/fallah20a.pdf)\n6. [Reptile: a Scalable Metalearning Algorithm / On First-Order Meta-Learning Algorithms](https://arxiv.org/pdf/1803.02999)\n7. [On the Convergence of Multi-Step MAML (NSF PAR copy)](https://par.nsf.gov/servlets/purl/10339904)\n8. [Survey of meta-learning landmarks (arXiv, 2026)](https://www.arxiv.org/pdf/2602.19837)\n9. [Alpha MAML: Adaptive Model-Agnostic Meta-Learning (alphaXiv overview)](https://arxiv.org/abs/1905.07435)\n10. [Andrychowicz, Marcin and colleagues (2016). Learning to learn by gradient descent by gradient descent. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1606.04474)\n11. [Vinyals, Oriol and colleagues (2016). Matching Networks for One Shot Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1606.04080)\n12. [Understanding transfer learning and gradient-based meta-learning techniques (Machine Learning journal, 2023)](https://link.springer.com/article/10.1007/s10994-023-06387-w)\n13. [Song, Xingyou and colleagues (2019). ES-MAML: Simple Hessian-Free Meta Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1910.01215)\n14. [Grant et al., Gradient-Based Meta-Learning as Hierarchical Bayesian Inference (LLAMA)](https://cocosci.princeton.edu/papers/gradient-based_meta-learning.pdf)\n15. [Bayesian Model-Agnostic Meta-Learning (BMAML, NeurIPS 2018)](https://proceedings.neurips.cc/paper/2018/file/e1021d43911ca2c1845910d84f40aeae-Paper.pdf)\n16. [Finn, Chelsea, Xu, Kelvin, Levine, Sergey (2018). Probabilistic Model-Agnostic Meta-Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1806.02817)\n17. [Abbas, Momin and colleagues (2022). Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2206.03996)\n18. [Min, Sewon and colleagues (2021). MetaICL: Learning to Learn In Context. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2110.15943)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods › Ensemble, boosting, and transfer methods › Transfer learning and domain adaptation*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Oct 15, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/model-agnostic-meta-learning", "markdown_url": "https://www.edgepedia.org/model-agnostic-meta-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Model-agnostic meta-learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/model-agnostic-meta-learning. Edgepedia Community License 1.0.", "credit_md": "\"[Model-agnostic meta-learning](https://www.edgepedia.org/model-agnostic-meta-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/model-agnostic-meta-learning](https://www.edgepedia.org/model-agnostic-meta-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Model-agnostic meta-learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/model-agnostic-meta-learning. Edgepedia Community License 1.0.", "speakable": "Model-agnostic meta-learning is a meta-learning algorithm, introduced by Chelsea Finn and colleagues in 2017, that trains initial parameters for fast adaptation to new tasks." }