# GSM8K
GSM8K (Grade School Math 8K) is a benchmark of 8,500 grade-school math word problems released by OpenAI in October 2021 to diagnose why language models fail at multi-step mathematical reasoning, and it became the standard yardstick for chain-of-thought reasoning before saturating and drawing contamination criticism in the mid-2020s.[1](https://arxiv.org/html/2110.14168v2)
| Key fact | Detail |
|---|---|
| Size | Roughly 8,500 problems: 7,473 training, 1,319 test (the split counts sum to 8,792), MIT licensed[2](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md) |
| Problem format | 2 to 8 arithmetic steps, natural-language solutions ending in a numeric answer after a '####' delimiter[1](https://arxiv.org/html/2110.14168v2) • [2](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md) |
| Built by | OpenAI (Karl Cobbe, Vineet Kosaraju, John Schulman, among others), data collected with Surge AI[3](https://openai.com/index/solving-math-word-problems/) |
| First headline result | Verifier system scored 55% (vendor-reported) against 60% for a sample of 9-12 year olds, October 2021[3](https://openai.com/index/solving-math-word-problems/) |
| Saturation | GPT-4 crossed 90% in March 2023; frontier models cluster at 96-99% by 2025-2026[4](https://itdoeswhatnow.com/benchmarks/gsm8k/) • [5](https://futureagi.com/glossary/gsm8k/) |
| Contamination | Problems appear verbatim in post-2021 tutorials; GSM1k showed accuracy drops of up to 8% on held-out equivalents[6](https://benchlist.ai/articles/gsm8k) • [7](https://doi.org/10.48550/arxiv.2405.00332) |
| Label noise | GSM8K-Platinum removed 110 flawed test questions and corrected 10 mislabeled answers in 2025[8](https://gradientscience.org/gsm8k-platinum/) |
| Status (2026) | Footnote and regression smoke test on model cards; AIME and FrontierMath differentiate frontier capability[5](https://futureagi.com/glossary/gsm8k/) |
## What GSM8K is
GSM8K was introduced in the paper "Training Verifiers to Solve Math Word Problems" as a dataset to diagnose the failures of current models and support research on multi-step mathematical reasoning.[1](https://arxiv.org/html/2110.14168v2) Each problem is a short word problem solvable in between 2 and 8 steps using basic arithmetic, written by human problem writers.[1](https://arxiv.org/html/2110.14168v2) The canonical split is 7,473 training and 1,319 test problems, and the dataset is MIT licensed.[2](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md)
Solutions are written in natural language with calculator annotations, ending in a final numeric answer after a '####' delimiter.[2](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md) OpenAI stated that fine-tuned state-of-the-art language models performed poorly on the dataset, primarily due to the high diversity of problems.[3](https://openai.com/index/solving-math-word-problems/)
## How it was built and scored
[Data collection](https://www.edgepedia.org/data-collection) began with Upwork freelancers and was scaled up with [Surge AI](https://www.edgepedia.org/surge-ai), the annotation vendor credited by OpenAI with performing the GSM8K data collection.[2](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md) • [3](https://openai.com/index/solving-math-word-problems/) [Quality control](https://www.edgepedia.org/quality-control) included an agreement-check round in which contractors re-solved problems; 1.7% of problems still produced disagreements among contractors, which the dataset card estimates as the fraction containing breaking errors or ambiguities.[2](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md)
Scoring is last-number exact match: the grader extracts the last number from the model's output (after stripping commas) and compares it to the canonical answer to within 1e-3, with canonical decoding at temperature 0.0 and max_tokens 512.[6](https://benchlist.ai/articles/gsm8k) Two scoring conventions exist, strict numeric match and tolerant text match, and most modern reports use strict.[6](https://benchlist.ai/articles/gsm8k)
Reported numbers vary for the same model for several reasons. Vendors evaluate with the prompt that maximizes each model's performance, whereas independent evaluations use a standardized setting, so independent GSM8K numbers may fall below vendor-reported figures.[7](https://doi.org/10.48550/arxiv.2405.00332) The 95%+ scores from frontier models also depend on chain-of-thought: without a scratchpad the same models drop more than 20 percentage points, and calculator-augmented models routinely add 8-15 percentage points over the same model without tools.[6](https://benchlist.ai/articles/gsm8k) Last-number grading is brittle as well: a model that says "we have 12 apples plus 5 oranges totalling 17" is scored as 17 (correct) when the question asked for apples (12).[6](https://benchlist.ai/articles/gsm8k)
## Chain of thought and the rise of GSM8K
GSM8K became the canonical reasoning anchor because it was the benchmark on which the reasoning techniques of 2021-2022 were first demonstrated. The original OpenAI paper found that finetuning a 6B model to output only the final answer without intermediate steps dropped performance drastically, from 20.6% to 5.2%, an early signal that intermediate reasoning steps were essential.[1](https://arxiv.org/html/2110.14168v2) OpenAI's verifier system, which generated 100 candidate solutions at test time and selected the one ranked highest by the verifier, scored 55% (vendor-reported) on problems where a small sample of 9-12 year olds scored 60%; the company reported this as nearly twice the accuracy of a fine-tuned GPT-3 model.[3](https://openai.com/index/solving-math-word-problems/) The paper calculated that the verifier boost was approximately equivalent to a 30x model size increase over finetuning alone, and that a 6B model with verification slightly outperformed a finetuned 175B model.[1](https://arxiv.org/html/2110.14168v2)
[Chain-of-thought prompting](https://www.edgepedia.org/chain-of-thought-prompting) then transformed the leaderboard. A single prompt phrase, "Let's think step by step", with no worked examples, lifted GSM8K accuracy from 10.4% to 40.7% in the May 2022 [zero-shot chain-of-thought](https://www.edgepedia.org/zero-shot-chain-of-thought) result.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/) PaLM 540B with 8-shot chain-of-thought reached 56.9% in January 2022.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/) These results established GSM8K as the standard "does this model do basic math" test for roughly four years.[6](https://benchlist.ai/articles/gsm8k)
## By the numbers
The score timeline, with vendor-reported figures kept distinct from independent ones:
- **October 2021**: GPT-3 175B fine-tuned with a verifier, about 55% (vendor-reported).[3](https://openai.com/index/solving-math-word-problems/)
- **January 2022**: PaLM 540B, 8-shot chain-of-thought, 56.9%.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/)
- **March 2023**: GPT-4, 92.0%, flagged by OpenAI with an asterisk because a portion of the GSM8K training set had leaked into GPT-4's pre-training mix.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/)
- **June 2024**: [Claude 3.5 Sonnet](https://www.edgepedia.org/claude-3-5-sonnet), 96.4%.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/)
- **July 2024**: [Llama 3.1 405B](https://www.edgepedia.org/llama-3-1-405b), 96.8% with 8-shot chain-of-thought.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/)
- **2025-2026**: frontier models cluster at 96-99%; a 2026 comparison lists GPT-5.x, Claude Opus 4.7, and [Gemini 3 Pro](https://www.edgepedia.org/gemini-3-pro) at about 99%, Llama 4 405B and Qwen 3 at about 98%, and Mistral Large 3 at about 97%.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/) • [5](https://futureagi.com/glossary/gsm8k/)
Models crossed 90% with GPT-4 in March 2023 and effectively saturated the benchmark by 2025, when top models routinely reported above 95%.[7](https://doi.org/10.48550/arxiv.2405.00332) • [6](https://benchlist.ai/articles/gsm8k) Sources describe the saturation level differently: the MadryLab team wrote in March 2025 that frontier performance had plateaued around 95%, while a 2026 guide states saturation above 98% on every frontier model.[8](https://gradientscience.org/gsm8k-platinum/) • [5](https://futureagi.com/glossary/gsm8k/)
## Contamination, gaming and label noise
**Contamination is well documented.** GSM8K problems and solutions appear verbatim in countless online tutorials published after 2021, so most modern training mixes contain them.[6](https://benchlist.ai/articles/gsm8k) OpenAI itself flagged the leakage into GPT-4's pre-training mix with an asterisk on the 92.0% score.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/)
The clearest test of contamination came from GSM1k, a May 2024 benchmark from [Scale AI](https://www.edgepedia.org/scale-ai) and the Center for AI Safety that mirrors GSM8K across human solve rates, number of solution steps, and answer magnitude, and was guaranteed not to be in model training data.[7](https://doi.org/10.48550/arxiv.2405.00332) Evaluating leading open- and closed-source models, the study observed accuracy drops of up to 8%, with several model families showing systematic overfitting across almost all model sizes.[7](https://doi.org/10.48550/arxiv.2405.00332) One journalism account describes some open models scoring up to 13 points lower on GSM1k than on GSM8K; the peer-reviewed paper's figure is drops of up to 8%.[4](https://itdoeswhatnow.com/benchmarks/gsm8k/) • [7](https://doi.org/10.48550/arxiv.2405.00332) The study also found a Spearman rank correlation of 0.36 (p = 0.03) between a model's per-character log-likelihood of generating GSM8k examples and its GSM8k-to-GSM1k performance gap, suggesting partial memorization.[7](https://doi.org/10.48550/arxiv.2405.00332) The work was peer-reviewed and published in the NeurIPS 2024 Datasets and Benchmarks track.[9](https://papers.nips.cc/paper_files/paper/2024/file/53384f2090c6a5cac952c598fd67992f-Paper-Datasets_and_Benchmarks_Track.pdf)
**Gaming was concentrated outside the frontier.** Lesser-known models near the top of the [Open LLM Leaderboard](https://www.edgepedia.org/open-llm-leaderboard) performed substantially worse on GSM1k, which the authors read as evidence of [Goodhart's law](https://www.edgepedia.org/goodharts-law): gaming the GSM8k benchmark rather than improving reasoning.[7](https://doi.org/10.48550/arxiv.2405.00332) The same study found that all frontier models showed minimal signs of overfitting and broadly generalized to novel math problems guaranteed not to be in their training data.[7](https://doi.org/10.48550/arxiv.2405.00332)
**Label noise inflated the plateau.** In work released with arXiv:2502.03461 (February 5, 2025), the MadryLab team flagged 219 GSM8K test questions, manually inspected them, removed 110, verified 99, and corrected 10 mislabeled answers, producing GSM8K-Platinum, a 1,209-example drop-in replacement for the 1,319-question test set (new annotations licensed CC BY-SA 4.0).[8](https://gradientscience.org/gsm8k-platinum/) • [10](https://huggingface.co/datasets/madrylab/gsm8k-platinum) They attribute much of the apparent ~95% performance plateau to label noise rather than actual model failures.[8](https://gradientscience.org/gsm8k-platinum/) The cleaning revealed real differences the original benchmark obscured: on GSM8K-Platinum, Claude 3.7 Sonnet (extended thinking) made 2 errors versus Llama 405B's 17, while on original GSM8K both showed 45 errors each.[8](https://gradientscience.org/gsm8k-platinum/)
## How it compares with other math benchmarks
By 2025, MATH-500 and AIME had replaced GSM8K as the benchmarks used to differentiate frontier capability, with GSM8K considered effectively saturated and used mainly as a smoke test.[6](https://benchlist.ai/articles/gsm8k) A 2026 comparison illustrates the separation: on GSM8K the frontier cluster sits at roughly 97-99%, but the same models spread widely on AIME 2025 (55-88%) and [FrontierMath](https://www.edgepedia.org/frontiermath) (5-22%).[5](https://futureagi.com/glossary/gsm8k/) GSM8K still differentiates a 60% model from a 90% model, but not a 95% model from a 97% model, a gap below the noise floor of the last-number grader.[6](https://benchlist.ai/articles/gsm8k)
## Recent developments
Three shifts define GSM8K's status between late 2023 and September 2026. First, saturation: frontier performance plateaued around 95% by early 2025 and reached roughly 97-99% by 2026, so the benchmark no longer separates top models.[8](https://gradientscience.org/gsm8k-platinum/) • [5](https://futureagi.com/glossary/gsm8k/) Second, demotion: recent frontier releases including o1 and [Claude 3.7 Sonnet](https://www.edgepedia.org/claude-3-7-sonnet) excluded GSM8K evaluations from their model reports in favor of more challenging benchmarks, and by 2026 it appears on model cards as a footnote rather than a headline.[8](https://gradientscience.org/gsm8k-platinum/) • [5](https://futureagi.com/glossary/gsm8k/) Third, persistence: the dataset remained remarkably popular, with over 350,000 downloads in February 2025 alone on [Hugging Face](https://www.edgepedia.org/hugging-face).[8](https://gradientscience.org/gsm8k-platinum/) Practitioners now use it as a regression smoke test, where a fine-tuned model dropping to 80% signals broken base behavior, and run contamination probes such as entity-renaming and quantity-renumbering, in which models that memorized solutions sometimes re-output the original answer despite changed inputs.[5](https://futureagi.com/glossary/gsm8k/) GSM8K-Platinum serves as the drop-in replacement where a cleaner signal is needed.[10](https://huggingface.co/datasets/madrylab/gsm8k-platinum)
## References
1. [Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, October 2021)](https://arxiv.org/html/2110.14168v2)
2. [openai/gsm8k dataset card (Hugging Face)](https://huggingface.co/datasets/openai/gsm8k/blob/main/README.md)
3. [Solving math word problems (OpenAI announcement, October 29, 2021)](https://openai.com/index/solving-math-word-problems/)
4. [The GSM8K AI benchmark, explained (It Does What Now?)](https://itdoeswhatnow.com/benchmarks/gsm8k/)
5. [What Is GSM8K? Definition & FutureAGI Guide (2026)](https://futureagi.com/glossary/gsm8k/)
6. [GSM8K, methodology, history, and how to verify a published score (Benchlist)](https://benchlist.ai/articles/gsm8k)
7. [A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k, Scale AI/CAIS, May 2024)](https://doi.org/10.48550/arxiv.2405.00332)
8. [GSM8K-Platinum: Revealing Performance Gaps in Frontier LLMs (MadryLab, March 6, 2025)](https://gradientscience.org/gsm8k-platinum/)
9. [GSM1k paper, NeurIPS 2024 Datasets and Benchmarks Track](https://papers.nips.cc/paper_files/paper/2024/file/53384f2090c6a5cac952c598fd67992f-Paper-Datasets_and_Benchmarks_Track.pdf)
10. [madrylab/gsm8k-platinum dataset card](https://huggingface.co/datasets/madrylab/gsm8k-platinum)
---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Evaluation, benchmarks, and leaderboards › Text, knowledge, and coding benchmarks*
*Initially written Sep 17, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026; Oct 11, 2026; Oct 15, 2026 · Last review: Sep 30, 2026*
*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*
License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license