{ "id": "epqvm2vsvq", "slug": "realm-machine-learning", "title": "REALM (machine learning)", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.modern_ai", "label": "Modern AI: foundation models, generative AI, and the AI industry", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai" }, { "id": "technology.computing.modern_ai.methods", "label": "Foundation-model methods and training", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.methods" } ], "geo": [ { "id": "geo.us.t2001.technology.computing.modern_ai.methods", "label": "United States · 2001 to 2020: Foundation-model methods and training", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai.methods", "path": [ { "id": "geo.us", "label": "United States", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us" }, { "id": "geo.us.t2001", "label": "United States · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001" }, { "id": "geo.us.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology" }, { "id": "geo.us.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing" }, { "id": "geo.us.t2001.technology.computing.modern_ai", "label": "Modern AI: foundation models, generative AI, and the AI industry", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai" }, { "id": "geo.us.t2001.technology.computing.modern_ai.methods", "label": "Foundation-model methods and training", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai.methods" } ] } ], "excerpt": "REALM (Retrieval-Augmented Language Model pre-training) is a method from Google Research, introduced at ICML 2020, that integrates a neural knowledge retriever into a language model so it retrieves documents before each prediction.", "snippet": "REALM (Retrieval-Augmented Language Model pre-training) is a method from Google Research, introduced at ICML 2020, that integrates a neural knowledge retriever into a language model so it retrieves documents before each prediction.", "node": "technology.computing.modern_ai.methods", "markdown": "# REALM (machine learning)\n\n**REALM** (Retrieval-Augmented Language Model pre-training) is a method for training language models in which a neural knowledge retriever is built into the model itself, so that before each prediction the model retrieves documents from a large corpus such as [Wikipedia](https://www.edgepedia.org/wikipedia) and attends over them, rather than storing all world knowledge implicitly in its parameters. It was introduced by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang of Google Research in a February 2020 arXiv preprint and a paper at the 37th [International Conference on Machine Learning](https://www.edgepedia.org/international-conference-on-machine-learning) (ICML 2020).[1](https://proceedings.mlr.press/v119/guu20a.html) • [2](https://arxiv.org/abs/2002.08909)\n\n| Key fact | Detail |\n|---|---|\n| Full name and venue | Retrieval-Augmented Language Model Pre-Training; ICML 2020, PMLR 119:3929-3938; arXiv 10 February 2020[1](https://proceedings.mlr.press/v119/guu20a.html) • [2](https://arxiv.org/abs/2002.08909) |\n| Authors | Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang (Google Research)[1](https://proceedings.mlr.press/v119/guu20a.html) |\n| Core idea | A latent knowledge retriever is used during pre-training, fine-tuning, and inference; the retriever is trained unsupervised with masked language modeling as the learning signal[1](https://proceedings.mlr.press/v119/guu20a.html) |\n| Training objective | Treat retrieved documents as latent variables and maximize the marginal likelihood of masked tokens, approximated over the top k documents found by Maximum Inner Product Search[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf) |\n| Headline result | New state of the art on three Open-QA benchmarks, outperforming all previous systems by 4-16% absolute accuracy[1](https://proceedings.mlr.press/v119/guu20a.html) |\n| Engineering challenge | The search index goes stale after each gradient update, so it is asynchronously refreshed by re-embedding and re-indexing all documents every several hundred training steps[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf) |\n| Later standing | On Natural Questions, REALM scored 40.4 exact match, later surpassed by DPR (41.5), RAG-Sequence (44.5), and FiD (51.4)[4](https://metaintelligencesystems.com/research/paper-retrieval-augmented-generation/) • [5](https://www.researchgate.net/publication/364527497_Decoupled_Context_Processing_for_Context_Augmented_Language_Modeling) |\n\n## Motivation: implicit versus explicit knowledge\n\nLanguage models pre-trained on large text corpora, such as BERT, RoBERTa, and T5, store a surprising amount of world knowledge in their parameters. The paper's motivating example is that BERT can correctly predict the missing word in \"The ___ is the currency of the United Kingdom\" (answer: \"pound\"). This knowledge is implicit: it is difficult to determine what is stored in the network and where, and storage space is limited by network size, so covering more facts requires ever-larger networks, which can be prohibitively slow or expensive to train.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\nREALM makes knowledge explicit. Before each prediction, the model retrieves documents from a textual knowledge corpus and attends over them to inform its output. The authors describe the qualitative benefits as interpretability and modularity: the retrieved documents expose which knowledge the model used, and the corpus can be changed without retraining the network.[1](https://proceedings.mlr.press/v119/guu20a.html) • [3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n## How it works\n\n**Retrieve, then predict.** REALM decomposes the probability of an output into two steps. Given an input x, a retriever defines a distribution p(z | x) over documents z in a knowledge corpus Z; a knowledge-augmented encoder then defines p(y | z, x), the probability of the output y given both the input and the retrieved document. The retrieved document is treated as a latent variable, and the overall likelihood marginalizes over documents:\n\n\\[ p(y \\mid x) = \\sum_{z \\in Z} p(y \\mid z, x)\\, p(z \\mid x) \\]\n\nFor pre-training, x is a sentence with tokens masked out and y is the missing tokens (the masked language modeling task popularized by BERT); for fine-tuning on open-domain question answering, x is a question and y is the answer string.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n**The retriever.** The retriever is a dense inner-product model: input and document are each mapped to d-dimensional vectors by BERT-style Transformer encoders, and the relevance score f(x, z) is the inner product of the two embeddings, with the retrieval distribution a softmax over all relevance scores. Documents are embedded from their title and body.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n**The knowledge-augmented encoder.** The input and the retrieved document's body are joined into a single sequence and fed into a separate [Transformer](https://www.edgepedia.org/transformer), allowing rich cross-attention between the question and the document before predicting the output. For pre-training, the encoder predicts each masked token with the standard MLM loss; for question answering, it assumes the answer appears as a contiguous span in the retrieved document and scores candidate spans with start and end token representations passed through a feed-forward network.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n**Training signal.** The key intuition is to train the retriever with a performance-based signal from unsupervised text: a retrieval that improves the language model's perplexity is helpful and should be rewarded, while an uninformative retrieval should be penalized. Because both components are differentiable, the gradient of the log-likelihood flows all the way back through the retrieval step. The paper states this is the first demonstration of pre-training a knowledge retriever in an unsupervised manner.[1](https://proceedings.mlr.press/v119/guu20a.html) • [3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n## Computational challenges\n\nThe marginal likelihood involves a sum over every document in the corpus, millions of candidates per training step. REALM approximates it by summing only over the top k documents under p(z | x), which is reasonable when most documents have near-zero probability. Finding the top k documents is formulated as Maximum Inner Product Search (MIPS), which runs in time and storage that scale sub-linearly with the number of documents; this requires pre-computing document embeddings and building a search index over them.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\nBecause the retriever's parameters change with every gradient update, the pre-computed index immediately goes stale. REALM's solution is to asynchronously refresh the index, re-embedding and re-indexing all documents every several hundred training steps. Between refreshes the index is slightly stale, but it is used only to select the top k candidates; the probabilities and gradients are recomputed exactly for those candidates.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n## Evaluation on open-domain question answering\n\nThe authors evaluate on open-domain question answering, a task they describe as among the most knowledge-intensive in natural language processing: given a question such as \"What is the currency of the UK?\", the model must output the answer string without being given a pre-identified document containing the answer, so it must draw on knowledge from millions of documents.[3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\nREALM was fine-tuned on three benchmarks, NaturalQuestions-Open, WebQuestions, and CuratedTREC, and compared against both extremely large models that store knowledge implicitly (such as T5) and earlier retrieval-based approaches that implement retrieval in a more heuristic fashion. It achieved new state-of-the-art results on all three, outperforming all previous systems by 4-16% absolute accuracy.[1](https://proceedings.mlr.press/v119/guu20a.html) • [3](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n\n## Place in the retrieval-augmented literature\n\nREALM appeared two months before Dense Passage Retrieval (DPR) and the Lewis et al. RAG paper, and retrospective surveys group the three as foundational works of retrieval-augmented generation. On [Natural Questions](https://www.edgepedia.org/natural-questions) exact match, later reported comparisons place REALM at 40.4, DPR at 41.5, RAG-Sequence at 44.5, and [FiD](https://www.edgepedia.org/fusion-in-decoder) at 51.4, with closed-book T5-11B at 34.5; REALM is listed at 110M parameters in one comparison table, against 400M for RAG and 770M for FiD.[4](https://metaintelligencesystems.com/research/paper-retrieval-augmented-generation/) • [5](https://www.researchgate.net/publication/364527497_Decoupled_Context_Processing_for_Context_Augmented_Language_Modeling) • [6](https://www.bloss0m.com/en/paper-reading/31-retrieval-augmented-generation/)\n\nThe distinctive cost of REALM's design is joint training: because the retriever is learned during pre-training, the document index must be refreshed as parameters change, a bill that DPR's authors argued could be avoided by training the retriever with supervised question-passage data and keeping the document encoder frozen.[6](https://www.bloss0m.com/en/paper-reading/31-retrieval-augmented-generation/) • [7](https://zhaifeiyue.github.io/papers/2005.11401/detail.html) Subsequent work pushed end-to-end retriever training further: Sachan and colleagues (ACL 2021) reported a top-20 retrieval accuracy of 84 on Natural Questions, a 5-point improvement over DPR, and answer-extraction results outperforming REALM and RAG by 3+ points.[8](https://arxiv.org/abs/2101.00408) A 2022 evaluation of retriever-augmented models including REALM examined their reasoning over retrieved statements, finding that multihop retrieve-and-read was promising for large language models like GPT-3.5 but did not generalize to models like Flan-T5-xxl.[9](https://arxiv.org/abs/2212.09146)\n\nThe code was released by Google Research in the google-research/language repository.[10](https://github.com/google-research/language/blob/master/language/realm/README.md)\n\n## References\n\n1. [Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M. (2020). Retrieval Augmented Language Model Pre-Training. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3929-3938.](https://proceedings.mlr.press/v119/guu20a.html)\n2. [Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909.](https://arxiv.org/abs/2002.08909)\n3. [REALM: Retrieval-Augmented Language Model Pre-Training (full paper PDF), PMLR.](https://proceedings.mlr.press/v119/guu20a/guu20a.pdf)\n4. [\"Retrieval-Augmented Generation\" - what it actually said, Meta Intelligence Systems.](https://metaintelligencesystems.com/research/paper-retrieval-augmented-generation/)\n5. [Decoupled Context Processing for Context Augmented Language Modeling, ResearchGate.](https://www.researchgate.net/publication/364527497_Decoupled_Context_Processing_for_Context_Augmented_Language_Modeling)\n6. [RAG: Attach Retrieval to Generation, but Do Not Treat 2020 RAG as a Production RAG Platform, Bloss0m paper reading](https://www.bloss0m.com/en/paper-reading/31-retrieval-augmented-generation/)\n7. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, detailed notes, Feiyue KB.](https://zhaifeiyue.github.io/papers/2005.11401/detail.html)\n8. [Sachan, D. S., et al. (2021). End-to-End Training of Neural Retrievers for Open-Domain Question Answering. arXiv:2101.00408.](https://arxiv.org/abs/2101.00408)\n9. [Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model. arXiv:2212.09146.](https://arxiv.org/abs/2212.09146)\n10. [REALM README, google-research/language, GitHub.](https://github.com/google-research/language/blob/master/language/realm/README.md)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training*\n\n*Initially written Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/realm-machine-learning", "markdown_url": "https://www.edgepedia.org/realm-machine-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"REALM (machine learning)\", Edgepedia (EdgeChat), https://www.edgepedia.org/realm-machine-learning. Edgepedia Community License 1.0.", "credit_md": "\"[REALM (machine learning)](https://www.edgepedia.org/realm-machine-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/realm-machine-learning](https://www.edgepedia.org/realm-machine-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"REALM (machine learning)\", Edgepedia (EdgeChat), https://www.edgepedia.org/realm-machine-learning. Edgepedia Community License 1.0.", "speakable": "REALM is a method from Google Research, introduced at ICML 2020, that integrates a neural knowledge retriever into a language model so it retrieves documents before each prediction." }