# REALM (machine learning)

**REALM** (Retrieval-Augmented Language Model pre-training) is a method for training language models in which a neural knowledge retriever is built into the model itself, so that before each prediction the model retrieves documents from a large corpus such as [Wikipedia](https://www.edgepedia.org/wikipedia) and attends over them, rather than storing all world knowledge implicitly in its parameters. It was introduced by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang of Google Research in a February 2020 arXiv preprint and a paper at the 37th [International Conference on Machine Learning](https://www.edgepedia.org/international-conference-on-machine-learning) (ICML 2020).<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup><sup> • </sup><sup>[2](https://arxiv.org/abs/2002.08909)</sup>

| Key fact | Detail |
|---|---|
| Full name and venue | Retrieval-Augmented Language Model Pre-Training; ICML 2020, PMLR 119:3929-3938; arXiv 10 February 2020<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup><sup> • </sup><sup>[2](https://arxiv.org/abs/2002.08909)</sup> |
| Authors | Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang (Google Research)<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup> |
| Core idea | A latent knowledge retriever is used during pre-training, fine-tuning, and inference; the retriever is trained unsupervised with masked language modeling as the learning signal<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup> |
| Training objective | Treat retrieved documents as latent variables and maximize the marginal likelihood of masked tokens, approximated over the top k documents found by Maximum Inner Product Search<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup> |
| Headline result | New state of the art on three Open-QA benchmarks, outperforming all previous systems by 4-16% absolute accuracy<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup> |
| Engineering challenge | The search index goes stale after each gradient update, so it is asynchronously refreshed by re-embedding and re-indexing all documents every several hundred training steps<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup> |
| Later standing | On Natural Questions, REALM scored 40.4 exact match, later surpassed by DPR (41.5), RAG-Sequence (44.5), and FiD (51.4)<sup>[4](https://metaintelligencesystems.com/research/paper-retrieval-augmented-generation/)</sup><sup> • </sup><sup>[5](https://www.researchgate.net/publication/364527497_Decoupled_Context_Processing_for_Context_Augmented_Language_Modeling)</sup> |

## Motivation: implicit versus explicit knowledge

Language models pre-trained on large text corpora, such as BERT, RoBERTa, and T5, store a surprising amount of world knowledge in their parameters. The paper's motivating example is that BERT can correctly predict the missing word in "The ___ is the currency of the United Kingdom" (answer: "pound"). This knowledge is implicit: it is difficult to determine what is stored in the network and where, and storage space is limited by network size, so covering more facts requires ever-larger networks, which can be prohibitively slow or expensive to train.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

REALM makes knowledge explicit. Before each prediction, the model retrieves documents from a textual knowledge corpus and attends over them to inform its output. The authors describe the qualitative benefits as interpretability and modularity: the retrieved documents expose which knowledge the model used, and the corpus can be changed without retraining the network.<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup><sup> • </sup><sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

## How it works

**Retrieve, then predict.** REALM decomposes the probability of an output into two steps. Given an input x, a retriever defines a distribution p(z | x) over documents z in a knowledge corpus Z; a knowledge-augmented encoder then defines p(y | z, x), the probability of the output y given both the input and the retrieved document. The retrieved document is treated as a latent variable, and the overall likelihood marginalizes over documents:

\[ p(y \mid x) = \sum_{z \in Z} p(y \mid z, x)\, p(z \mid x) \]

For pre-training, x is a sentence with tokens masked out and y is the missing tokens (the masked language modeling task popularized by BERT); for fine-tuning on open-domain question answering, x is a question and y is the answer string.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

**The retriever.** The retriever is a dense inner-product model: input and document are each mapped to d-dimensional vectors by BERT-style Transformer encoders, and the relevance score f(x, z) is the inner product of the two embeddings, with the retrieval distribution a softmax over all relevance scores. Documents are embedded from their title and body.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

**The knowledge-augmented encoder.** The input and the retrieved document's body are joined into a single sequence and fed into a separate [Transformer](https://www.edgepedia.org/transformer), allowing rich cross-attention between the question and the document before predicting the output. For pre-training, the encoder predicts each masked token with the standard MLM loss; for question answering, it assumes the answer appears as a contiguous span in the retrieved document and scores candidate spans with start and end token representations passed through a feed-forward network.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

**Training signal.** The key intuition is to train the retriever with a performance-based signal from unsupervised text: a retrieval that improves the language model's perplexity is helpful and should be rewarded, while an uninformative retrieval should be penalized. Because both components are differentiable, the gradient of the log-likelihood flows all the way back through the retrieval step. The paper states this is the first demonstration of pre-training a knowledge retriever in an unsupervised manner.<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup><sup> • </sup><sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

## Computational challenges

The marginal likelihood involves a sum over every document in the corpus, millions of candidates per training step. REALM approximates it by summing only over the top k documents under p(z | x), which is reasonable when most documents have near-zero probability. Finding the top k documents is formulated as Maximum Inner Product Search (MIPS), which runs in time and storage that scale sub-linearly with the number of documents; this requires pre-computing document embeddings and building a search index over them.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

Because the retriever's parameters change with every gradient update, the pre-computed index immediately goes stale. REALM's solution is to asynchronously refresh the index, re-embedding and re-indexing all documents every several hundred training steps. Between refreshes the index is slightly stale, but it is used only to select the top k candidates; the probabilities and gradients are recomputed exactly for those candidates.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

## Evaluation on open-domain question answering

The authors evaluate on open-domain question answering, a task they describe as among the most knowledge-intensive in natural language processing: given a question such as "What is the currency of the UK?", the model must output the answer string without being given a pre-identified document containing the answer, so it must draw on knowledge from millions of documents.<sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

REALM was fine-tuned on three benchmarks, NaturalQuestions-Open, WebQuestions, and CuratedTREC, and compared against both extremely large models that store knowledge implicitly (such as T5) and earlier retrieval-based approaches that implement retrieval in a more heuristic fashion. It achieved new state-of-the-art results on all three, outperforming all previous systems by 4-16% absolute accuracy.<sup>[1](https://proceedings.mlr.press/v119/guu20a.html)</sup><sup> • </sup><sup>[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)</sup>

## Place in the retrieval-augmented literature

REALM appeared two months before Dense Passage Retrieval (DPR) and the Lewis et al. RAG paper, and retrospective surveys group the three as foundational works of retrieval-augmented generation. On [Natural Questions](https://www.edgepedia.org/natural-questions) exact match, later reported comparisons place REALM at 40.4, DPR at 41.5, RAG-Sequence at 44.5, and [FiD](https://www.edgepedia.org/fusion-in-decoder) at 51.4, with closed-book T5-11B at 34.5; REALM is listed at 110M parameters in one comparison table, against 400M for RAG and 770M for FiD.<sup>[4](https://metaintelligencesystems.com/research/paper-retrieval-augmented-generation/)</sup><sup> • </sup><sup>[5](https://www.researchgate.net/publication/364527497_Decoupled_Context_Processing_for_Context_Augmented_Language_Modeling)</sup><sup> • </sup><sup>[6](https://www.bloss0m.com/en/paper-reading/31-retrieval-augmented-generation/)</sup>

The distinctive cost of REALM's design is joint training: because the retriever is learned during pre-training, the document index must be refreshed as parameters change, a bill that DPR's authors argued could be avoided by training the retriever with supervised question-passage data and keeping the document encoder frozen.<sup>[6](https://www.bloss0m.com/en/paper-reading/31-retrieval-augmented-generation/)</sup><sup> • </sup><sup>[7](https://zhaifeiyue.github.io/papers/2005.11401/detail.html)</sup> Subsequent work pushed end-to-end retriever training further: Sachan and colleagues (ACL 2021) reported a top-20 retrieval accuracy of 84 on Natural Questions, a 5-point improvement over DPR, and answer-extraction results outperforming REALM and RAG by 3+ points.<sup>[8](https://arxiv.org/abs/2101.00408)</sup> A 2022 evaluation of retriever-augmented models including REALM examined their reasoning over retrieved statements, finding that multihop retrieve-and-read was promising for large language models like GPT-3.5 but did not generalize to models like Flan-T5-xxl.<sup>[9](https://arxiv.org/abs/2212.09146)</sup>

The code was released by Google Research in the google-research/language repository.<sup>[10](https://github.com/google-research/language/blob/master/language/realm/README.md)</sup>

## References

1. [Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M. (2020). Retrieval Augmented Language Model Pre-Training. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3929-3938.](https://proceedings.mlr.press/v119/guu20a.html)
2. [Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909.](https://arxiv.org/abs/2002.08909)
3. [REALM: Retrieval-Augmented Language Model Pre-Training (full paper PDF), PMLR.](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)
4. ["Retrieval-Augmented Generation" - what it actually said, Meta Intelligence Systems.](https://metaintelligencesystems.com/research/paper-retrieval-augmented-generation/)
5. [Decoupled Context Processing for Context Augmented Language Modeling, ResearchGate.](https://www.researchgate.net/publication/364527497_Decoupled_Context_Processing_for_Context_Augmented_Language_Modeling)
6. [RAG: Attach Retrieval to Generation, but Do Not Treat 2020 RAG as a Production RAG Platform, Bloss0m paper reading](https://www.bloss0m.com/en/paper-reading/31-retrieval-augmented-generation/)
7. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, detailed notes, Feiyue KB.](https://zhaifeiyue.github.io/papers/2005.11401/detail.html)
8. [Sachan, D. S., et al. (2021). End-to-End Training of Neural Retrievers for Open-Domain Question Answering. arXiv:2101.00408.](https://arxiv.org/abs/2101.00408)
9. [Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model. arXiv:2212.09146.](https://arxiv.org/abs/2212.09146)
10. [REALM README, google-research/language, GitHub.](https://github.com/google-research/language/blob/master/language/realm/README.md)

---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training*

*Initially written Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license