# KL regularization in RLHF

KL regularization in reinforcement learning from human feedback (RLHF) is a penalty term, added to the RL objective, that keeps the fine-tuned language model close to a frozen reference policy by charging it the Kullback–Leibler (KL) divergence between the two. It is the central design knob of RL post-training pipelines: the coefficient that sets how much reward a policy must gain to justify changing its behavior.

| Key fact | Detail |
|---|---|
| Objective | Maximize expected reward minus \(\beta\) times KL divergence to a frozen reference policy: \(\max_{\pi} J(\theta) = \mathbb{E}[r(x,y)] - \beta \cdot \mathrm{KL}(\pi(\cdot \| x) \,\|\, \pi_{\mathrm{ref}}(\cdot \| x))\) <sup>[1](https://rlhfbook.com/c/15-regularization)</sup> |
| Reference policy | Most often the instruction-tuned (SFT) model; sometimes a previous RL checkpoint <sup>[1](https://rlhfbook.com/c/15-regularization)</sup> |
| Dominant form | Reverse KL to the reference, computed as a Monte Carlo estimate over generated tokens <sup>[1](https://rlhfbook.com/c/15-regularization)</sup> |
| Landmark numbers | InstructGPT used \(\lambda_{\mathrm{KL}} = 0.001\) with \(D_{\mathrm{KL}}\) typically below 100 nats, and the authors wrote the penalty might be wholly unnecessary <sup>[2](https://arxiv.org/pdf/2204.05862)</sup> |
| Purpose | Combating reward hacking and distribution collapse during RL fine-tuning <sup>[3](https://ar5iv.labs.arxiv.org/html/2205.11275)</sup><sup> • </sup><sup>[4](https://cs.mcgill.ca/~comp579/W24/Lectures/24-rlhf-2024.pdf)</sup> |
| Known tension | The KL trust region and PPO's ratio clipping can conflict, excluding high-reward policies outside their intersection <sup>[5](https://arxiv.org/html/2602.11523)</sup> |
| 2024–2026 shift | Multi-target and dynamic references, reference-free alignment, and elimination of reference regularization under verifiable rewards <sup>[5](https://arxiv.org/html/2602.11523)</sup> |

## What the KL penalty is

The canonical RLHF objective maximizes the expected reward of the policy's outputs minus a coefficient β times the KL divergence between the current policy and a frozen reference policy: \(\max_{\pi} J(\theta) = \mathrm{E}[r(x,y)] - \beta \cdot \mathrm{KL}(\pi(\cdot\|x) \,\|\, \pi_{\mathrm{ref}}(\cdot\|x))\) <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>. The KL divergence measures how much probability mass the fine-tuned model places where the reference model places little; it does not satisfy the formal properties of a true distance metric, but practitioners colloquially call it a "KL distance" <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>. β determines the trade-off: how much reward is needed to justify departing from the reference by a given distance <sup>[3](https://ar5iv.labs.arxiv.org/html/2205.11275)</sup>.

The reference policy is most often the instruction-tuned model, not the raw base model, though it can also be a previous RL checkpoint <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

The idea predates the LLM era. The same objective appeared as "conservative fine-tuning" (Jaques et al. 2017) and KL-control (Jaques et al. 2019), and a KL divergence penalty was applied to dialogue agents well before large language models became popular; it then underpinned the recipes of Ziegler et al. 2019, Stiennon et al. 2020, Perez et al. 2022, and Bai et al. 2022 <sup>[3](https://ar5iv.labs.arxiv.org/html/2205.11275)</sup><sup> • </sup><sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

## Why it exists: reward hacking, distribution collapse and drift

Unconstrained RL fine-tuning against a learned reward model has known failure modes. Korbak et al. frame the KL penalty as a solution to distribution collapse: preserving the distributional properties of the language model is folded into the objective as a KL term to the pretrained model <sup>[3](https://ar5iv.labs.arxiv.org/html/2205.11275)</sup>. Teaching material frames it as a penalty that prevents the policy from diverging too far from the pretrained model, and identifies reward hacking as a common problem in RL for chatbots <sup>[4](https://cs.mcgill.ca/~comp579/W24/Lectures/24-rlhf-2024.pdf)</sup>. Standard pipeline descriptions say the KL penalty against the initialization model combats reward hacking <sup>[5](https://arxiv.org/html/2602.11523)</sup>.

In short, β buys safety with reward: a larger β keeps outputs closer to the reference, at the cost of leaving reward on the table <sup>[3](https://ar5iv.labs.arxiv.org/html/2205.11275)</sup>.

## How it is implemented in practice

As of 2026, the most popular variant in RLHF implementations is a KL distance from the current policy to a reference policy across generated samples. The most common penalty is the reverse KL to the reference policy, computed in practice as a [Monte Carlo](https://www.edgepedia.org/monte-carlo) estimate: tokens are sampled from the RL model, and probabilities are computed from the reference model <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

InstructGPT shows the classic wiring: an empirically estimated per-token KL penalty is folded into the PPO reward, with total reward \(r_{\mathrm{total}} = r_{\mathrm{PM}} - \lambda_{\mathrm{KL}} \cdot D_{\mathrm{KL}}(\text{policy} \,\|\, \text{policy}_0)\), where policy_0 is the initial SFT policy and PPO (Schulman et al. 2017) stabilizes training, following Stiennon et al. 2020 <sup>[2](https://arxiv.org/pdf/2204.05862)</sup>. This means the pipeline carries two constraints at once: a KL penalty against the initialization model \(\pi_0\) to combat reward hacking, and policy ratio clipping against the current policy \(\pi_t\) for optimization stability <sup>[5](https://arxiv.org/html/2602.11523)</sup>. The reference model also adds a forward pass per generated sample, since its probabilities must be evaluated on the RL model's tokens <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

## By the numbers

The most prominent empirical numbers come from [InstructGPT](https://www.edgepedia.org/instructgpt). The authors used a very small KL coefficient, \(\lambda_{\mathrm{KL}} = 0.001\), and noted it likely had a very minor impact during most of RL training, since \(D_{\mathrm{KL}}\) was typically below 100 nats, and might actually be wholly unnecessary <sup>[2](https://arxiv.org/pdf/2204.05862)</sup>. That is a vendor-reported caveat from the paper itself, and it sits oddly against the penalty's status as the most popular regularization in most RLHF implementations as of 2026 <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

InstructGPT also found that \(\sqrt{D_{\mathrm{KL}}(\pi \| \pi_0)}\) and reward are approximately linearly related for much of RLHF training <sup>[2](https://arxiv.org/pdf/2204.05862)</sup>.

## Reverse KL, forward KL and alternatives

The choice of KL direction changes the geometry of optimization. Reverse KL promotes mode-seeking behavior, concentrating probability mass on dominant modes; forward KL (as in TRPO) encourages mode-covering behavior, where the policy attempts to match all modes of the reference <sup>[5](https://arxiv.org/html/2602.11523)</sup>. Reverse KL has a numerical property: it applies a large penalty when the new model puts substantial probability mass where the reference assigns low probability <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>. It also fits on-policy sampling, since the estimate only needs reference probabilities on tokens the RL model actually generates <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

Proposed alternatives target the penalty's blind spots. For heavy-tailed reward error, one proposal is to regularize by a function other than KL, such as capping the maximum odds ratio for any action (similar to quantilizers), or to make the reward a bounded function <sup>[6](https://www.lesswrong.com/posts/pEZoTSCxHY3mfPbHu/catastrophic-goodhart-in-rl-with-kl-penalty)</sup>. The 2026 unification paper proposes reverse KL to the current policy rather than the initialization model, differing from TRPO's forward-KL formulation <sup>[5](https://arxiv.org/html/2602.11523)</sup>.

## Recent developments

Three shifts define 2024–2026. First, research moved toward multi-target references and dynamic reference optimization (Gorbatovski et al. 2024; Ramé et al. 2024; Yuan et al. 2026), and toward reference-free alignment that replaces the KL term with SFT losses <sup>[5](https://arxiv.org/html/2602.11523)</sup>. Second, several lines of work demonstrated the feasibility of completely eliminating reference regularization under specific assumptions: uniform reference policies (Xu et al. 2024), length-controlled alignment (Gupta et al. 2024; Meng et al. 2024; Xiao et al. 2025), and verifiable rewards in RLVR settings (Yu et al. 2025) <sup>[5](https://arxiv.org/html/2602.11523)</sup>. Third, the framing changed: the 2026 unification paper argues reference regularization should be viewed not as universally mandatory but as a flexible, task-dependent design choice <sup>[5](https://arxiv.org/html/2602.11523)</sup>.

Despite this, the KL distance from generations to a reference remains the most popular regularization in most RLHF implementations as of 2026 <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>. Regularization techniques beyond the core KL distance are often used to stabilize experimental setups and then simplified away in the next model generation <sup>[1](https://rlhfbook.com/c/15-regularization)</sup>.

Likewise, no retrieved source gives typical β ranges beyond InstructGPT's 0.001, [Llama 2](https://www.edgepedia.org/llama-2)/3 KL budgets, head-to-head evaluations against DPO-family methods, or evidence on whether lower KL improves calibration or reduces hallucination.

## References

1. [Regularization | RLHF and Post-Training Book by Nathan Lambert](https://rlhfbook.com/c/15-regularization)
2. [Training language models to follow instructions with human feedback (InstructGPT, Ouyang et al., 2022)](https://arxiv.org/pdf/2204.05862)
3. [RL with KL penalties is better viewed as Bayesian inference (Korbak et al., 2022)](https://ar5iv.labs.arxiv.org/html/2205.11275)
4. [Reinforcement Learning for LLMs / RLHF (McGill COMP 579 lecture, Winter 2024)](https://cs.mcgill.ca/~comp579/W24/Lectures/24-rlhf-2024.pdf)
5. [Unifying Stable Optimization and Reference Regularization in RLHF (February 2026)](https://arxiv.org/html/2602.11523)
6. [Catastrophic Goodhart in RL with KL penalty (LessWrong)](https://www.lesswrong.com/posts/pEZoTSCxHY3mfPbHu/catastrophic-goodhart-in-rl-with-kl-penalty)

---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Post-training and alignment methods*

*Initially written Sep 17, 2026 · Reviewed: Sep 30, 2026 · Edited: Oct 11, 2026; Oct 15, 2026 · Last review: Sep 30, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license