{ "id": "eprmdp0aef", "slug": "reward-learning", "title": "Reward learning", "updated": "2026-09-30", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" } ], "geo": [ { "id": "geo.nongeo.t1946.technology.computing.ai.ml_methods", "label": "Non-geographic · 1946 to 2000: Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_methods", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology" }, { "id": "geo.nongeo.t1946.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing" }, { "id": "geo.nongeo.t1946.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_methods" } ] }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Non-geographic · 2001 to 2020: Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods" } ] } ], "excerpt": "Reward learning is a machine learning method that infers a reward function from human preferences, demonstrations, or ratings, and underlies RLHF fine-tuning of instruction-following language models like InstructGPT.", "snippet": "Reward learning is a machine learning method that infers a reward function from human preferences, demonstrations, or ratings, and underlies RLHF fine-tuning of instruction-following language models like InstructGPT.", "node": "technology.computing.ai.ml_methods", "markdown": "# Reward learning\n\nReward learning infers a reward function from data such as human preferences, demonstrations, or ratings, giving a reinforcement learning agent a learned objective in place of a hand-designed one. [Reinforcement learning from human feedback (RLHF)](https://www.edgepedia.org/reinforcement-learning-from-human-feedback-rlhf) combines three processes: feedback collection, reward modeling, and policy optimization against the learned reward model.[1](https://arxiv.org/abs/2307.15217) The approach first showed it could solve Atari games and simulated robot locomotion from pairwise human preferences on less than 1% of an agent's environment interactions,[2](https://papers.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf) and later became the fine-tuning method behind instruction-following language models such as [InstructGPT](https://www.edgepedia.org/instructgpt).[3](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)\n\n| Key fact | Value |\n|---|---|\n| What is learned | A scalar reward function, most commonly a Bradley-Terry model trained by maximum likelihood on preference pairs[4](https://rlhfbook.com/c/05-reward-models) |\n| RLHF pipeline | Feedback collection, reward model training, then policy optimization (typically PPO) with KL regularization[1](https://arxiv.org/abs/2307.15217) |\n| Human labeling cost (2017 robotics/Atari) | 15 minutes to 5 hours of non-expert feedback; a backflip behavior needed about 900 bits of feedback and under an hour of evaluator time[2](https://papers.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf) • [5](https://web.archive.org/web/20230618223746/https:/openai.com/research/learning-from-human-preferences) |\n| Headline LLM result | 1.3B InstructGPT outputs preferred over 175B GPT-3 outputs despite roughly 135 times fewer parameters[3](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf) |\n| Reward-model accuracy ceiling | 60-70% on prior RLHF validation sets, set by inter-annotator disagreement[6](https://aclanthology.org/2025.findings-naacl.96.pdf) |\n| Reward-model-free alternative | Direct preference optimization (DPO) optimizes the same KL-constrained objective without an explicit reward model or RL[7](https://proceedings.neurips.cc/paper%5Ffiles/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf) |\n\n## How it works\n\nThe canonical reward model is a learned scalar function \\( r_{\\theta} \\) trained from pairwise comparisons. Under the Bradley-Terry model, which the 2017 deep RL work described as the specialization of the Luce-Shephard choice rule to preferences over trajectory segments, the probability that a human prefers one segment depends exponentially on the latent reward summed over the roughly 1-2 second clip.[2](https://papers.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf) For language models, training is maximum likelihood: \\[ \\theta^{*} = \\arg\\min_{\\theta}\\; \\mathbb{E}_{(x, y_{\\mathrm{c}}, y_{\\mathrm{r}}) \\sim \\mathcal{D}}\\bigl[ -\\log \\sigma\\bigl( r_{\\theta}(y_{\\mathrm{c}} \\mid x) - r_{\\theta}(y_{\\mathrm{r}} \\mid x) \\bigr) \\bigr], \\] where \\( y_{\\mathrm{c}} \\) and \\( y_{\\mathrm{r}} \\) are the chosen and rejected completions; the reward difference is the log odds that one response is preferred.[4](https://rlhfbook.com/c/05-reward-models) • [3](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf) Identifiability has a clean condition: because the Bradley-Terry likelihood depends only on score differences, the model is identifiable only up to an additive constant, and under a location constraint such as sum-to-zero the maximum-likelihood estimate exists and is unique if and only if the comparison graph is strongly connected; if the graph is not strongly connected, a finite maximum-likelihood estimate may fail to exist or be nonunique depending on the graph.[8](https://mlhp.stanford.edu/src/chap2.html)\n\n## How it is done\n\nThe RLHF pipeline has three stages: supervised fine-tuning, preference sampling and reward learning, and reinforcement learning optimization against the reward model, with the objective \\[ \\max_{\\pi_{\\theta}}\\; \\mathbb{E}_{x \\sim \\mathcal{D},\\, y \\sim \\pi_{\\theta}(y \\mid x)}\\bigl[ r_{\\phi}(x,y) \\bigr] - \\beta\\, \\mathbb{D}_{\\mathrm{KL}}\\bigl[ \\pi_{\\theta} \\,\\|\\, \\pi_{\\mathrm{ref}} \\bigr]. \\][7](https://proceedings.neurips.cc/paper%5Ffiles/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf) In InstructGPT, a team of 40 contractors labeled demonstrations and comparisons, ranking between \\( K = 9 \\) responses per prompt, and the reward model trained on all \\( K \\)-choose-2 pairwise comparisons from each prompt as a single batch element. Only 6B reward models were used because 175B reward-model training was unstable and less suitable as the value function during RL, and a per-token KL penalty from the SFT model mitigated reward-model over-optimization.[3](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf) [Implementation](https://www.edgepedia.org/implementation) practice appends a small linear head to the language model to produce the scalar reward and trains for only 1 epoch to avoid overfitting.[4](https://rlhfbook.com/c/05-reward-models)\n\n## Origin\n\nRecovering an objective from observed behavior predates modern RL: the IRL literature credits Russell (1998) with the informal problem statement.[9](https://people.eecs.berkeley.edu/%7erussell/papers/ml00-irl.pdf) Ng and Russell (2000) formulated inverse reinforcement learning in Markov decision processes, extracting a reward function given observed optimal behavior, with three algorithms covering finite state spaces, linear function approximation, and finite trajectories, cast as a linear program; they also identified degeneracy, the existence of many reward functions consistent with the observed policy, as a key issue.[9](https://people.eecs.berkeley.edu/%7erussell/papers/ml00-irl.pdf) [Apprenticeship learning](https://www.edgepedia.org/apprenticeship-learning) guarantees a policy that performs as well as the expert on the expert's unknown reward without necessarily recovering it, demonstrated by mimicking five distinct driving styles in a highway simulator from 2-minute demonstrations.[10](http://ai.stanford.edu/%7Epabbeel/pubs/AbbeelNg_alvirl_ICML2004.pdf)\n\nPreference-based reward learning at deep RL scale was introduced by [Paul Christiano](https://www.edgepedia.org/paul-christiano) and colleagues in 2017 on arXiv,[11](https://doi.org/10.48550/arxiv.1706.03741) in a paper that fitted a reward model to human preferences while simultaneously training a policy (A2C for Atari, TRPO for robotics); learning a separate reward model via supervised learning reduced interaction complexity by roughly three orders of magnitude.[2](https://papers.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf) A 2018 arXiv follow-up by Borja Ibarz and colleagues combined demonstrations and trajectory preferences in Atari.[12](https://doi.org/10.48550/arxiv.1811.06521) The 2017 work established preference-based reward learning for deep RL and influenced later LLM alignment research, in which preference-based alignment gained prominence with InstructGPT in 2022, the connection to Bradley-Terry reward modeling was made explicit in InstructGPT by Long Ouyang and colleagues (2022, arXiv),[13](https://doi.org/10.48550/arxiv.2203.02155) • [8](https://mlhp.stanford.edu/src/chap2.html) and DPO by Rafael Rafailov and colleagues (2023, arXiv) reformulated RLHF as Bradley-Terry maximum likelihood without a separate reward model.[14](https://doi.org/10.48550/arxiv.2305.18290) • [8](https://mlhp.stanford.edu/src/chap2.html)\n\n## Variants\n\n**Inverse reinforcement learning (IRL)** learns a reward from demonstrations, which carry rich information but are harder to gather because they require more effort and expertise than comparisons.[1](https://arxiv.org/abs/2307.15217) Adversarial IRL (AIRL), from the 2017 arXiv paper by Justin Fu, Katie Luo, and [Sergey Levine](https://www.edgepedia.org/sergey-levine), is a scalable adversarial formulation that recovers rewards robust to changes in environment dynamics, in contrast to unmodified IRL methods that recover brittle rewards; GAIL, by comparison, does not recover reward functions at all.[15](https://users.cs.utah.edu/~dsbrown/readings/airl.pdf) • [16](https://doi.org/10.48550/arxiv.1710.11248)\n\n**RLHF** uses human preference comparisons; **RLAIF** substitutes AI feedback. Across summarization and helpful dialogue, RLAIF reached comparable human win rates over an SFT baseline and a higher harmlessness rate.[17](https://raw.githubusercontent.com/mlresearch/v235/main/assets/lee24t/lee24t.pdf) • [18](https://doi.org/10.48550/arxiv.2309.00267) Direct-RLAIF goes further and obtains rewards directly from an off-the-shelf LLM during RL, circumventing reward-model training and addressing reward-model staleness.[18](https://doi.org/10.48550/arxiv.2309.00267)\n\n**Direct preference optimization (DPO)** derives a maximum-likelihood objective from offline preference data formulated over the policy and a reference model, bypassing explicit reward model training and RL while remaining equivalent to the Bradley-Terry model with an implicit reward \\( r^{*}(y \\mid x) = \\beta \\log \\frac{\\pi_{\\theta}(y \\mid x)}{\\pi_{\\mathrm{ref}}(y \\mid x)} \\).[14](https://doi.org/10.48550/arxiv.2305.18290) • [19](https://arxiv.org/html/2410.15595v4) • [8](https://mlhp.stanford.edu/src/chap2.html) DPO spawned a family of variants, including [KTO](https://www.edgepedia.org/kahneman-tversky-optimization), IPO, CPO, ORPO, and SimPO, and standard offline DPO suffers a distributional discrepancy that has motivated online and iterative variants.[19](https://arxiv.org/html/2410.15595v4) **Reward modeling** as a standalone activity also spans process reward models, which score every step of a chain of thought, and [outcome reward models](https://www.edgepedia.org/outcome-reward-model), which predict correctness of a whole completion.[4](https://rlhfbook.com/c/05-reward-models)\n\n## Applications\n\nInstructGPT's reward-learned 1.3B model produced outputs preferred over the 175B GPT-3 baseline, and the same pipeline underlies instruction-following assistants.[3](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf) In RL proper, preference-based reward learning solved Atari games and simulated robot locomotion, including hard-to-specify behaviors such as a backflip and driving with the flow of traffic, from 15 minutes to 5 hours of non-expert feedback.[2](https://papers.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf) Apprenticeship learning via IRL reproduced five distinct driving styles in a highway driving simulator from short demonstrations.[10](http://ai.stanford.edu/%7Epabbeel/pubs/AbbeelNg_alvirl_ICML2004.pdf)\n\n## Limitations and alternatives\n\n**Reward hacking.** [Reward hacking](https://www.edgepedia.org/reward-hacking) is formally defined as the case where optimizing an imperfect proxy reward \\( \\tilde{\\mathcal{R}} \\) degrades performance under the true reward \\( \\mathcal{R} \\); for the set of all stochastic policies, two reward functions can only be unhackable if one of them is constant, so unhackable proxies are essentially unavailable in complex environments.[20](https://dl.acm.org/doi/10.5555/3600270.3600957) Reward models can misgeneralize into poor proxies even from correctly labeled data, and many reward functions fit the same feedback dataset even with infinite data; without KL regularization, LLMs undergoing RL learn to output nonsensical text.[1](https://arxiv.org/abs/2307.15217)\n\n**Overoptimization.** In a synthetic setup using a fixed 6B gold reward model from InstructGPT to label proxy reward models of 3M to 3B parameters, optimizing the proxy too much eventually hindered ground-truth performance, in line with [Goodhart's law](https://www.edgepedia.org/goodharts-law), with different functional forms for RL versus best-of-n sampling. RL is far less KL-efficient than best-of-n sampling, and the KL penalty acts like early stopping.[21](https://proceedings.mlr.press/v202/gao23h.html) • [22](https://doi.org/10.48550/arxiv.2210.10760)\n\n**Data quality ceilings.** Prior RLHF validation sets such as Anthropic's Helpful and Harmless data and OpenAI's Learning to Summarize have accuracy ceilings between 60 and 70% due to inter-annotator disagreement, so a reward model cannot be validated beyond that range on such data.[6](https://aclanthology.org/2025.findings-naacl.96.pdf) [Benchmarking](https://www.edgepedia.org/benchmarking) has matured in parallel: [RewardBench](https://www.edgepedia.org/rewardbench) comprises 1,755 prompt-chosen-rejected trios across chat, reasoning, and safety, where some subsets are solved at 100% accuracy by small reward models, while top-10 models exceed 95% on Chat but fall to 85–91% on Chat Hard, exposing weak detection of subtle prompt mismatches.[6](https://aclanthology.org/2025.findings-naacl.96.pdf) • [23](https://doi.org/10.48550/arxiv.2403.13787)\n\n**What the learned function estimates.** If human preferences arise from a regret model rather than the partial-return model used in InstructGPT, ChatGPT, Sparrow, and [Llama 2](https://www.edgepedia.org/llama-2), the function learned by standard preference-based reward learning approximates the optimal advantage function rather than a reward function; interpreting it as a reward is typically not ruinous to performance, which explains why partial-return RLHF works in practice.[24](https://ojs.aaai.org/index.php/AAAI/article/download/28870/29653)\n\n**The alternative reward learning replaces.** Hand-designed rewards are the baseline being displaced, and they carry their own failure modes: computational experiments and a controlled expert study show reward functions can be overfit to particular learning algorithms and their hyperparameters, and that experts' myopic per-state-action reward design leads to invalid task specifications.[25](https://ojs.aaai.org/index.php/AAAI/article/view/25733) A systematic study of reward misspecification in hand-designed rewards documented misaligned models across environments.[26](https://doi.org/10.48550/arxiv.2201.03544)\n\n**Reward modeling versus direct optimization.** One theoretical analysis under loglinear policies and linear rewards finds RLHF \\( \\Theta(\\sqrt{d_{R}/n}) \\)-close to its objective while DPO is \\( \\Theta(d_{P}/(\\beta n)) \\)-close, so DPO wins asymptotically for \\( n \\gg d \\) but RLHF has the smaller suboptimality gap when \\( n < d \\), the typical LLM regime.[27](https://ar5iv.labs.arxiv.org/html/2403.01857) Published comparisons do not fully agree on which approach is statistically more efficient in practice, and the debate remains open.\n\n## References\n\n1. [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://arxiv.org/abs/2307.15217)\n2. [Deep Reinforcement Learning from Human Preferences (NeurIPS 2017)](https://papers.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)\n3. [Training language models to follow instructions with human feedback (InstructGPT, NeurIPS 2022)](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)\n4. [Reward Modeling, RLHF Book (Nathan Lambert)](https://rlhfbook.com/c/05-reward-models)\n5. [Learning from human preferences (OpenAI blog, June 2017)](https://web.archive.org/web/20230618223746/https:/openai.com/research/learning-from-human-preferences)\n6. [RewardBench: Evaluating Reward Models for Language Modeling (NAACL 2025 Findings)](https://aclanthology.org/2025.findings-naacl.96.pdf)\n7. [Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023)](https://proceedings.neurips.cc/paper%5Ffiles/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf)\n8. [Machine Learning from Human Preferences, Chapter 2: Learning](https://mlhp.stanford.edu/src/chap2.html)\n9. [Algorithms for Inverse Reinforcement Learning (Ng & Russell, ICML 2000)](https://people.eecs.berkeley.edu/%7erussell/papers/ml00-irl.pdf)\n10. [Apprenticeship Learning via Inverse Reinforcement Learning (Abbeel & Ng, ICML 2004)](http://ai.stanford.edu/%7Epabbeel/pubs/AbbeelNg_alvirl_ICML2004.pdf)\n11. [Christiano, Paul and colleagues (2017). Deep reinforcement learning from human preferences. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1706.03741)\n12. [Ibarz, Borja and colleagues (2018). Reward learning from human preferences and demonstrations in Atari. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1811.06521)\n13. [Ouyang, Long and colleagues (2022). Training language models to follow instructions with human feedback. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2203.02155)\n14. [Rafailov, Rafael and colleagues (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2305.18290)\n15. [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (AIRL, ICLR 2018)](https://users.cs.utah.edu/~dsbrown/readings/airl.pdf)\n16. [Fu, Justin, Luo, Katie, Levine, Sergey (2017). Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1710.11248)\n17. [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024)](https://raw.githubusercontent.com/mlresearch/v235/main/assets/lee24t/lee24t.pdf)\n18. [Lee, Harrison and colleagues (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2309.00267)\n19. [A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://arxiv.org/html/2410.15595v4)\n20. [Defining and characterizing reward hacking (Skalse et al., NeurIPS 2023)](https://dl.acm.org/doi/10.5555/3600270.3600957)\n21. [Scaling Laws for Reward Model Overoptimization (Gao, Schulman, Hilton, ICML 2023)](https://proceedings.mlr.press/v202/gao23h.html)\n22. [Gao, Leo, Schulman, John, Hilton, Jacob (2022). Scaling Laws for Reward Model Overoptimization. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2210.10760)\n23. [Lambert, Nathan and colleagues (2024). RewardBench: Evaluating Reward Models for Language Modeling. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2403.13787)\n24. [Learning Optimal Advantage from Preferences and Mistaking It for Reward (AAAI 2024)](https://ojs.aaai.org/index.php/AAAI/article/download/28870/29653)\n25. [The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task Specifications (AAAI 2023)](https://ojs.aaai.org/index.php/AAAI/article/view/25733)\n26. [Pan, Alexander, Bhatia, Kush, Steinhardt, Jacob (2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2201.03544)\n27. [Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences](https://ar5iv.labs.arxiv.org/html/2403.01857)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://users.cs.utah.edu/~dsbrown/readings/airl.pdf" ], "url": "https://www.edgepedia.org/reward-learning", "markdown_url": "https://www.edgepedia.org/reward-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Reward learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/reward-learning. Edgepedia Community License 1.0.", "credit_md": "\"[Reward learning](https://www.edgepedia.org/reward-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/reward-learning](https://www.edgepedia.org/reward-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Reward learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/reward-learning. Edgepedia Community License 1.0.", "speakable": "Reward learning is a machine learning method that infers a reward function from human preferences, demonstrations, or ratings, and underlies RLHF fine-tuning of instruction-following language models like InstructGPT." }