{ "id": "epw6tejsba", "slug": "imitation-learning", "title": "Imitation learning", "updated": "2026-10-21", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" } ], "geo": [ { "id": "geo.nongeo.t1946.technology.computing.ai.ml_methods", "label": "Non-geographic · 1946 to 2000: Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_methods", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology" }, { "id": "geo.nongeo.t1946.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing" }, { "id": "geo.nongeo.t1946.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_methods" } ] }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Non-geographic · 2001 to 2020: Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods" } ] } ], "excerpt": "Imitation learning trains an agent from expert demonstrations by learning a mapping between observations and actions, without a hand-specified reward function, and is widely used in robotics and games.", "snippet": "Imitation learning trains an agent from expert demonstrations by learning a mapping between observations and actions, without a hand-specified reward function, and is widely used in robotics and games.", "node": "technology.computing.ai.ml_methods", "markdown": "# Imitation learning\n\nImitation learning trains an agent to perform a task from expert demonstrations by learning a mapping between observations and actions, instead of from a hand-specified reward function.[1](https://dl.acm.org/doi/10.1145/3054912) It sits between supervised learning and reinforcement learning: the simplest form, behavior cloning, is supervised learning on state-action pairs, while inverse reinforcement learning instead recovers the reward the expert appears to be optimizing.[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html) The approach is widely used in robotics and games.[1](https://dl.acm.org/doi/10.1145/3054912)\n\n| Key fact | Detail |\n|---|---|\n| Output | A stochastic policy giving an action distribution, \\( \\pi(a \\mid s) \\) (behavior cloning), or a reward function (inverse RL); a deterministic mapping from states to actions is a special case[3](https://arxiv.org/pdf/2404.19456v2.pdf) |\n| Core guarantee | Behavior cloning's return shortfall is Θ(εT²) for per-step error ε and horizon T, measured under the expert's state distribution; under the learner's own state distribution it is O(εT)[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html) |\n| Adversarial objective | GAIL minimizes the Jensen-Shannon divergence between the learner's and expert's occupancy measures[4](https://proceedings.neurips.cc/paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf) |\n| Typical data | Robomimic benchmark: demonstration counts vary by split, with 200 trajectories per task in the PH split and 300 per task in the MH split; Diffusion Policy reaches 100% success on Lift and Can from 200 human teleoperation demonstrations[5](https://www.roboticscenter.ai/datasets/robomimic) |\n| Compute | Benchmarking one new algorithm with BC-RNN and Diffusion Policy on the five Robomimic simulation tasks takes about 2 GPU-days[5](https://www.roboticscenter.ai/datasets/robomimic) |\n| Modern deployment | Supervised fine-tuning of language models is an instance of behavior cloning: context is the state, the next token is the action[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html) |\n\n## How it works\n\nThe formal setting is a [Markov decision process](https://www.edgepedia.org/markov-decision-process) without a reward function. A policy is a function mapping states to actions, and a demonstration is a state-action pair taken from a teacher's trajectory.[3](https://arxiv.org/pdf/2404.19456v2.pdf) Given demonstrations Ξ = {ξ₁, ..., ξ_D} from an expert policy π*, the problem is to find a policy that imitates \\( \\pi^{*} \\) without access to a reward.[6](https://web.stanford.edu/class/cs237b/pdfs/lecture/cs237b_lecture_12.pdf)\n\n[Behavior cloning](https://www.edgepedia.org/behavior-cloning) fits π_θ(a|s) to the demonstrated pairs by maximizing log-likelihood, that is, minimizing cross-entropy; it needs neither a transition model nor a reward.[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html) Divergence-minimizing variants instead match the occupancy measure, the distribution over states and actions the policy induces: GAIL's objective finds the policy whose occupancy measure minimizes the Jensen-Shannon divergence to the expert's, minus an entropy bonus.[4](https://proceedings.neurips.cc/paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf) Inverse RL runs the arrow of optimal control backwards: for a linear reward R(x,u) = wᵀφ(x,u), reward weights are estimated by maximum likelihood over the demonstrations.[6](https://web.stanford.edu/class/cs237b/pdfs/lecture/cs237b_lecture_12.pdf)\n\nTheory quantifies the compounding problem: with per-step disagreement probability ε under the expert's state distribution, behavior cloning's return shortfall is Θ(εT²), while measured under the learner's own state distribution it is \\( O(\\varepsilon T) \\).[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html)\n\n## How it is done\n\nA practitioner first collects demonstrations, typically by teleoperation or virtual-reality recording; the Robomimic PH split is 200 demonstrations from a single expert teleoperator, and one diffusion-based study used 566 VR-collected kitchen trajectories.[5](https://www.roboticscenter.ai/datasets/robomimic) • [7](https://arxiv.org/pdf/2301.10677) State-action pairs are extracted, a model class is chosen, and the policy is trained by supervised likelihood maximization.[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html)\n\nEvaluation is by closed-loop rollout success rate, averaged over 50 rollouts per task in [Robomimic](https://www.edgepedia.org/robomimic).[5](https://www.roboticscenter.ai/datasets/robomimic) If the policy drifts off the demonstrated distribution, an interactive correction round can be added: DAgger rolls out the current policy, asks the expert to label the visited states, aggregates the new labeled data into the dataset, and retrains.[8](https://algorithmsbook.com/files/chapter-18.pdf)\n\n## Origin\n\nThe earliest well-known system is ALVINN, a three-layer back-propagation network for road following on CMU's NAVLAB vehicle.[9](https://proceedings.neurips.cc/paper_files/paper/1988/file/812b4ba287f5ee0bc9d43bbf5bbe87fb-Paper.pdf) Dean A. Pomerleau described efficient training of neural networks for autonomous navigation in a 1991 Neural Computation paper.[10](https://doi.org/10.1162/neco.1991.3.1.88) The inverse problem has older roots in control theory: P. Moylan and B. Anderson's 1973 IEEE Transactions on Automatic Control paper studied nonlinear regulator theory and an inverse optimal control problem.[11](https://doi.org/10.1109/tac.1973.1100365) The term inverse reinforcement learning comes from Andrew Y. Ng and [Stuart Russell](https://www.edgepedia.org/stuart-j-russell)'s 2000 paper \"Algorithms for Inverse Reinforcement Learning\", published at ICML '00.[12](https://dl.acm.org/doi/10.1145/1015330.1015430) Later milestones include DAgger, reported by Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell in 2010 on arXiv as a reduction of imitation learning to no-regret online learning[13](https://doi.org/10.1184/r1/6550949), and generative adversarial imitation learning, reported by Jonathan Ho and [Stefano Ermon](https://www.edgepedia.org/stefano-ermon) in 2016 on arXiv.[14](https://doi.org/10.48550/arxiv.1606.03476)\n\n## Variants\n\n**Interactive methods** query the expert during training. DAgger iterates policy rollouts, expert relabeling, and retraining on the aggregate dataset; an earlier family of algorithms slowly mixes the learner's policy from the expert's toward the learned one.[13](https://doi.org/10.1184/r1/6550949) • [15](https://proceedings.mlr.press/v9/ross10a.html)\n\n**Adversarial methods** match behavior with a discriminator. GAIL alternates an Adam step on the discriminator with a TRPO policy step, the discriminator acting as a local cost.[4](https://proceedings.neurips.cc/paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf) A model-based variant uses a forward model to make the computation fully differentiable, enabling exact discriminator gradients.[16](https://proceedings.mlr.press/v70/baram17a.html) Discriminator-Actor-Critic (DAC) uses a modified GAIL discriminator to recover a dynamics-robust reward function.[17](https://arxiv.org/pdf/1809.02925)\n\n**Reward-recovery and matching methods** avoid adversarial training. IQ-Learn learns a single Q-function implicitly representing both reward and policy.[18](https://proceedings.neurips.cc/paper/2021/file/210f760a89db30aa72ca258a3483cc7f-Paper.pdf) SQIL, introduced by Siddharth Reddy, Anca D. Dragan, and [Sergey Levine](https://www.edgepedia.org/sergey-levine) in 2019 on arXiv, casts imitation as RL with sparse rewards.[19](https://doi.org/10.48550/arxiv.1905.11108) A 2021 game-theoretic framework by Gokul Swamy and colleagues classifies algorithms as reward-matching or action-value-matching and yields three templates, AdVIL, AdRIL, and DAeQuIL.[20](https://doi.org/10.48550/arxiv.2103.03236) Further extensions include goal-conditioned GAIL combined with hindsight relabeling[21](https://proceedings.neurips.cc/paper/2019/file/c8d3a760ebab631565f8509d84b3b3f1-Paper.pdf), imitation from observations without expert actions[22](https://ar5iv.labs.arxiv.org/html/2309.02473), and the model-based MILO framework of Jonathan Chang and colleagues, which uses offline data from a suboptimal behavior policy and needs coverage only of the expert's traces.[23](https://doi.org/10.48550/arxiv.2106.03207)\n\n**Diffusion and transformer policies** model the full multimodal action distribution while avoiding GAN instability.[7](https://arxiv.org/pdf/2301.10677) Action chunking with transformers predicts future action chunks in a single forward pass for low-latency control, at lower inference cost than iterative diffusion denoising.[24](https://www.tandfonline.com/doi/full/10.1080/01691864.2026.2726690)\n\n## Applications\n\nRobot manipulation is the main benchmarking ground: Robomimic's five task suites (Lift, Can, Square, Transport, ToolHang) simulated in robosuite/MuJoCo.[5](https://www.roboticscenter.ai/datasets/robomimic) Autonomous driving was the original application, from ALVINN's road following[9](https://proceedings.neurips.cc/paper_files/paper/1988/file/812b4ba287f5ee0bc9d43bbf5bbe87fb-Paper.pdf) to an apprenticeship learning driving simulation in which expert features from a single 1200-sample trajectory sufficed to mimic five driving styles at 25 m/s.[12](https://dl.acm.org/doi/10.1145/1015330.1015430) Games served as early testbeds for DAgger, including Super Tux Kart steering and Super Mario Bros[13](https://doi.org/10.1184/r1/6550949), and diffusion-based behavior cloning has modeled human gameplay in [Counter-Strike: Global Offensive](https://www.edgepedia.org/counter-strike-global-offensive).[7](https://arxiv.org/pdf/2301.10677) In language modeling, supervised fine-tuning on curated responses is behavior cloning applied to next-token prediction.[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html)\n\n## Limitations and alternatives\n\n**Covariate shift** is the central failure mode: the learner trains on expert states but is tested on states its own actions induce, and once it drifts into out-of-distribution states it cannot return to demonstrated ones.[22](https://ar5iv.labs.arxiv.org/html/2309.02473) A FrozenLake experiment makes this concrete: a cloned policy with zero error on demonstration pairs attains about one quarter of the expert's success rate.[2](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html) Demonstrations are also typically suboptimal, noisy, and multimodal, and unimodal models force researchers toward small or heavily curated datasets; GAIL is mode-seeking and suffers mode collapse, capturing only a subset of behaviors.[25](https://www.dfki.de/fileadmin/user_upload/import/16768_2408.04380v3.pdf) • [26](https://arxiv.org/pdf/1707.02747v2.pdf) Adversarial rewards carry a bias: positive reward \\( -\\log(1 - D(s,a)) \\) pushes agents toward survival, negative reward \\( \\log(D(s,a)) \\) toward early termination.[27](https://proceedings.mlr.press/v222/arulkumaran24a/arulkumaran24a.pdf)\n\nAgainst alternatives, the comparison is empirical rather than settled. On Robomimic, BC-RNN on PH data matches or beats every offline RL method tested on every task.[5](https://www.roboticscenter.ai/datasets/robomimic) Yet offline RL on sufficiently noisy suboptimal data can beat behavior cloning given expert data, especially on long-horizon problems, with conservative offline RL attaining Õ(√H) suboptimality.[28](https://ar5iv.labs.arxiv.org/html/2204.05618) A common remedy is a behavior cloning phase followed by reinforcement learning fine-tuning.[25](https://www.dfki.de/fileadmin/user_upload/import/16768_2408.04380v3.pdf) [Evaluation](https://www.edgepedia.org/evaluation) itself lacks standardization, making comparisons between approaches difficult.[3](https://arxiv.org/pdf/2404.19456v2.pdf)\n\n## References\n\n1. [Imitation Learning: A Survey of Learning Methods](https://dl.acm.org/doi/10.1145/3054912)\n2. [Learning from Demonstrations – Dive into Deep Learning](https://d2l.smola.org/chapter_reinforcement-learning/imitation.html)\n3. [A Survey of Imitation Learning Methods, Environments and Metrics](https://arxiv.org/pdf/2404.19456v2.pdf)\n4. [Generative Adversarial Imitation Learning (Ho & Ermon, NeurIPS 2016)](https://proceedings.neurips.cc/paper_files/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf)\n5. [Robomimic: Canonical Imitation Learning Benchmark](https://www.roboticscenter.ai/datasets/robomimic)\n6. [Imitation Learning (Stanford CS237B lecture notes)](https://web.stanford.edu/class/cs237b/pdfs/lecture/cs237b_lecture_12.pdf)\n7. [Diffusion models as observation-to-action models for behaviour cloning (arXiv 2301.10677, 'Diffusion BC')](https://arxiv.org/pdf/2301.10677)\n8. [Imitation Learning (Algorithms for Decision Making, Ch. 18)](https://algorithmsbook.com/files/chapter-18.pdf)\n9. [ALVINN: An Autonomous Land Vehicle in a Neural Network](https://proceedings.neurips.cc/paper_files/paper/1988/file/812b4ba287f5ee0bc9d43bbf5bbe87fb-Paper.pdf)\n10. [Dean A. Pomerleau (1991). Efficient Training of Artificial Neural Networks for Autonomous Navigation. Neural Computation.](https://doi.org/10.1162/neco.1991.3.1.88)\n11. [P. Moylan, B. Anderson (1973). Nonlinear regulator theory and an inverse optimal control problem. IEEE Transactions on Automatic Control.](https://doi.org/10.1109/tac.1973.1100365)\n12. [Apprenticeship learning via inverse reinforcement learning](https://dl.acm.org/doi/10.1145/1015330.1015430)\n13. [Ross, Stephane, Gordon, Geoffrey J., Bagnell, J. Andrew (2010). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. arXiv (Cornell University).](https://doi.org/10.1184/r1/6550949)\n14. [Ho, Jonathan, Ermon, Stefano (2016). Generative Adversarial Imitation Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1606.03476)\n15. [Efficient Reductions for Imitation Learning](https://proceedings.mlr.press/v9/ross10a.html)\n16. [End-to-End Differentiable Adversarial Imitation Learning (Baram, Anschel, Caspi, Mannor, ICML 2017)](https://proceedings.mlr.press/v70/baram17a.html)\n17. [Discriminator-Actor-Critic (DAC) (arXiv 1809.02925)](https://arxiv.org/pdf/1809.02925)\n18. [IQ-Learn: Inverse soft-Q Learning for Imitation (NeurIPS 2021)](https://proceedings.neurips.cc/paper/2021/file/210f760a89db30aa72ca258a3483cc7f-Paper.pdf)\n19. [Reddy, Siddharth, Dragan, Anca D., Levine, Sergey (2019). SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1905.11108)\n20. [Swamy, Gokul and colleagues (2021). Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2103.03236)\n21. [Goal-conditioned Imitation Learning / goalGAIL (NeurIPS 2019)](https://proceedings.neurips.cc/paper/2019/file/c8d3a760ebab631565f8509d84b3b3f1-Paper.pdf)\n22. [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://ar5iv.labs.arxiv.org/html/2309.02473)\n23. [Chang, Jonathan D. and colleagues (2021). Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2106.03207)\n24. [Hybrid-ACT: scaling robot imitation learning under few-demonstration settings with vector-quantized cross-attention](https://www.tandfonline.com/doi/full/10.1080/01691864.2026.2726690)\n25. [A Survey on Learning from Multimodal Demonstrations with Deep Generative Models](https://www.dfki.de/fileadmin/user_upload/import/16768_2408.04380v3.pdf)\n26. [Robust Imitation of Diverse Behaviors (arXiv 1707.02747)](https://arxiv.org/pdf/1707.02747v2.pdf)\n27. [A Pragmatic Look at Deep Imitation Learning](https://proceedings.mlr.press/v222/arulkumaran24a/arulkumaran24a.pdf)\n28. [When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?](https://ar5iv.labs.arxiv.org/html/2204.05618)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026; Oct 21, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/imitation-learning", "markdown_url": "https://www.edgepedia.org/imitation-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Imitation learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/imitation-learning. Edgepedia Community License 1.0.", "credit_md": "\"[Imitation learning](https://www.edgepedia.org/imitation-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/imitation-learning](https://www.edgepedia.org/imitation-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Imitation learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/imitation-learning. Edgepedia Community License 1.0.", "speakable": "Imitation learning trains an agent from expert demonstrations by learning a mapping between observations and actions, without a hand-specified reward function, and is widely used in robotics and games." }