# RefinedWeb
RefinedWeb is an English-only pretraining dataset of roughly five trillion tokens (word fragments; the units LLMs are trained on), built by the [Technology Innovation Institute](https://www.edgepedia.org/technology-innovation-institute) (TII) from heavily filtered and deduplicated [Common Crawl](https://www.edgepedia.org/common-crawl) web data and announced in June 2023.[1](https://arxiv.org/html/2306.01116v1) It was produced with TII's MDR (MacroData Refinement) pipeline and served two purposes: as the data foundation of the [Falcon](https://www.edgepedia.org/falcon-model-family) large language models, and as a demonstration that curated Common Crawl alone can match the curated data mixtures used by other frontier labs.[1](https://arxiv.org/html/2306.01116v1) The paper was peer-reviewed and published in the NeurIPS 2023 Datasets and Benchmarks track.[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets_and_Benchmarks.pdf)
| Fact | Value |
|---|---|
| Full dataset size | ~5 trillion tokens, ~10 billion documents, from Common Crawl only[1](https://arxiv.org/html/2306.01116v1) |
| Languages | English-only dataset; 58.20% of processed Common Crawl identified as English across 176 languages[1](https://arxiv.org/html/2306.01116v1) |
| Public release | ~600B-token extract (968M pages, 2.8TB unpacked), ODC-By 1.0 license[1](https://arxiv.org/html/2306.01116v1) • [3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) |
| Total data removed | ~75.4% by combined filtering and deduplication; ~50% of lines by deduplication alone[1](https://arxiv.org/html/2306.01116v1) |
| Benchmark effect | Small-agg score rose from 52.7% to 56.2% after filtering plus deduplication[1](https://arxiv.org/html/2306.01116v1) |
| Models trained on it | Falcon-7B/40B (with curated corpora); Falcon-RW-1B/7B on 350B tokens of RefinedWeb alone[3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) |
| Peer review | NeurIPS 2023 Datasets and Benchmarks track[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets_and_Benchmarks.pdf) |
## What RefinedWeb is
RefinedWeb was created by the Technology Innovation Institute, a privately funded research entity, built on top of Common Crawl, the public periodic crawls of the web.[1](https://arxiv.org/html/2306.01116v1) The full dataset contains approximately 10 billion documents, or around 5 trillion tokens, all derived from web pages rather than books, papers, or code.[1](https://arxiv.org/html/2306.01116v1) Although the dataset is described as English-focused, language identification over the processed Common Crawl classified content into 176 languages, with 58.20% of documents identified as English.[1](https://arxiv.org/html/2306.01116v1)
The language distribution only partially matches the worldwide distribution of speakers. The paper reports Russian as over-represented, [Mandarin Chinese](https://www.edgepedia.org/mandarin-chinese) under-represented, and Hindi absent from the top 20 languages despite being the third most spoken language worldwide.[1](https://arxiv.org/html/2306.01116v1)
## The curation pipeline (MDR)
MDR applies a staged sequence of filters and deduplication steps. According to the dataset card, the pipeline first filters URLs to remove adult content using a blocklist and a score system, then extracts text from pages with trafilatura, performs language identification with the fastText classifier from CCNet, and applies MassiveWeb heuristics and line-wise corrections, before running URL, fuzzy, and exact substring deduplication.[3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
Deduplication, not quality filtering, does most of the work. The paper states that exact substring matching plus MinHash fuzzy deduplication removed about 50% of lines, and that the team used strict settings leading to removal rates far higher than others have reported.[1](https://arxiv.org/html/2306.01116v1) Combined filtering and deduplication removed about 75.4% of the raw data overall.[1](https://arxiv.org/html/2306.01116v1)
The stated philosophy was "neutral filtering": scale first, strict deduplication, and no machine-learning-based quality filtering beyond language identification, with URL filtering used only for adult content.[1](https://arxiv.org/html/2306.01116v1)
## The central claim: web-only data suffices
The paper's central claim is that properly filtered and deduplicated web data alone can produce powerful models, significantly outperforming state-of-the-art models trained on [The Pile](https://www.edgepedia.org/the-pile), a widely used curated mixture of books, papers, code, and web text.[1](https://arxiv.org/html/2306.01116v1) The authors reported that models trained on RefinedWeb matched the performance of the GPT-3 series on zero-shot benchmarks using web data alone, even though the high-quality sources used in The Pile were excluded from RefinedWeb.[1](https://arxiv.org/html/2306.01116v1)
The evidence came from controlled training runs. TII trained Falcon-RW-1B and Falcon-RW-7B on 350 billion tokens of RefinedWeb alone to demonstrate its quality against curated corpora.[3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) Falcon-RW-1B is a 1B-parameter causal decoder-only model with a GPT-3-style architecture using ALiBi, tokenized with the GPT-2 tokenizer, and trained on English data only.[4](https://huggingface.co/keras/falcon_refinedweb_1b_en)
The ablations behind the claim separated the two curation levers. Deduplication delivered a steady performance boost across all datasets tested, and removal rates correlated with performance changes; The Pile and OSCAR-22.01 were found to contain 40-60% duplicates, while C4 and OSCAR-21.09 were already well deduplicated.[1](https://arxiv.org/html/2306.01116v1) Filtering improvements, by contrast, were not systematic across datasets. Combined filtering and deduplication raised RefinedWeb's small-agg benchmark score from 52.7% to 56.2%.[1](https://arxiv.org/html/2306.01116v1)
## Release, licensing and availability
TII publicly released a 600-billion-token extract of RefinedWeb together with 1.3B and 7.5B parameter models trained on it, and stated in the paper's FAQ that it did not plan to share the full 5 trillion-token dataset.[1](https://arxiv.org/html/2306.01116v1) The NeurIPS poster abstract describes the released extract as 500 billion tokens, a discrepancy with the paper's 600B figure.[5](https://nips.cc/virtual/2023/poster/73487)
The public extract, hosted as falcon-refinedweb on [Hugging Face](https://www.edgepedia.org/hugging-face), contains about 1 billion instances (968,000,015 individual web pages) totaling 2.8TB of clean text, roughly 500GB to download, and should contain 500-650 billion tokens depending on the tokenizer used.[3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) It represents about a tenth of the full dataset and was built from all Common Crawl dumps up to the 2023-06 one.[1](https://arxiv.org/html/2306.01116v1) It is released under an ODC-By 1.0 license, with users also required to abide by the Common Crawl Terms of Use.[3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) A third-party comparison characterizes RefinedWeb as therefore only partially closed: the recipe was published, but only ~600B tokens of the data were.[6](https://zeroentropy.dev/concepts/fineweb/)
## By the numbers
The paper's comparison table places RefinedWeb far ahead of earlier open corpora in scale: C4 at ~360GT, OSCAR-21.09 at ~370GT, OSCAR-22.01 at ~283GT, GPT-3's training data at ~300GT, The Pile at ~340GT, PaLM's at ~780GT, and RefinedWeb at ~5,000GT, with only RefinedWeb's public extract fully open.[1](https://arxiv.org/html/2306.01116v1) The argument was that scale plus strict deduplication of a single source could substitute for mixing many curated sources.
The removal figures quantify how aggressive the curation was. Deduplication alone removed about 50% of lines; filtering plus deduplication together removed 75.4% of the raw crawl, leaving a quarter of the original text.[1](https://arxiv.org/html/2306.01116v1) The benchmark gain from that removal was 3.5 points on the small-agg aggregate (52.7% to 56.2%).[1](https://arxiv.org/html/2306.01116v1)
## Use in Falcon and later comparisons
RefinedWeb was used in conjunction with curated corpora to train Falcon-7B and Falcon-40B, TII's flagship open models of 2023, and alone for the Falcon-RW demonstration models.[3](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
In 2024, Hugging Face's FineWeb began as a fully open replication of RefinedWeb, with the full dataset released under the ODC-By 1.0 license that RefinedWeb's full version lacked.[7](https://huggingface.co/datasets/HuggingFaceFW/fineweb) FineWeb consists of more than 18.5T tokens (originally counted at 15 trillion with the GPT-2 tokenizer) of cleaned, deduplicated English web data from 96 Common Crawl dumps covering summer 2013 to April 2024.[7](https://huggingface.co/datasets/HuggingFaceFW/fineweb) According to the FineWeb card, models trained on it outperform models trained on C4, Dolma-v1.6, The Pile, SlimPajama, and RedPajama-2, and additional filtering steps pushed its performance above RefinedWeb itself.[7](https://huggingface.co/datasets/HuggingFaceFW/fineweb)
Independent fixed-compute measurement points the same way: at 1B-parameter, fixed-compute training, FineWeb-Edu beats RefinedWeb by 4-8 points on average across MMLU and other benchmarks.[6](https://zeroentropy.dev/concepts/fineweb/) FineWeb-2, released in late 2024, extended the approach to 1000+ languages, though English remains 60%+ of total tokens and per-language token counts are uneven.[6](https://zeroentropy.dev/concepts/fineweb/) These results partly confirm and partly supersede the RefinedWeb claim: open web-only corpora do work, but the specific 2023 recipe has been surpassed by classifier-filtered successors.
## References
1. [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (arXiv, 2023)](https://arxiv.org/html/2306.01116v1)
2. [The RefinedWeb Dataset for Falcon LLM (NeurIPS 2023 Datasets and Benchmarks)](https://proceedings.neurips.cc/paper_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets_and_Benchmarks.pdf)
3. [tiiuae/falcon-refinedweb · Hugging Face dataset card](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
4. [keras/falcon_refinedweb_1b_en · Hugging Face model card](https://huggingface.co/keras/falcon_refinedweb_1b_en)
5. [NeurIPS 2023 Poster: The RefinedWeb Dataset for Falcon LLM](https://nips.cc/virtual/2023/poster/73487)
6. [FineWeb: HuggingFace's web corpus (ZeroEntropy)](https://zeroentropy.dev/concepts/fineweb/)
7. [HuggingFaceFW/fineweb · Hugging Face dataset card](https://huggingface.co/datasets/HuggingFaceFW/fineweb)
---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Pretraining data and corpora*
*Initially written Sep 17, 2026 · Edited: Oct 11, 2026; Oct 15, 2026*
*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*
License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license