53 articles
AGIEval
AGIEval is a bilingual benchmark for evaluating foundation models, built from 8,062 questions drawn from official exams like the SAT, Gaokao, and law admission tests.
Aider LLM Leaderboards
The Aider LLM Leaderboards are independent coding benchmarks built by Paul Gauthier around his Aider pair-programming tool, ranking large language models on editing code until unit tests pass.
AIME (LLM evaluation)
AIME, the American Invitational Mathematics Examination, is a high-school math contest that since September 2024 has doubled as a headline benchmark for large language model reasoning.
AlpacaEval
AlpacaEval is an automatic, LLM-judged benchmark and leaderboard for instruction-following chat models, built at Stanford and released in 2023, best known for its length-controlled win rate.
APPS (benchmark)
APPS (Automated Programming Progress Standard) is a benchmark for code generation, introduced at NeurIPS 2021, with 10,000 Python problems judged by running generated code against test cases.
ARC benchmark
The ARC benchmark (AI2 Reasoning Challenge) is a question-answering evaluation of 7,787 grade-school science questions released by the Allen Institute for AI in 2018, split into Easy and Challenge sets.
ARC Prize
The ARC Prize is an annual international competition on Kaggle, run by the nonprofit ARC Prize Foundation, awarding cash prizes for progress on the ARC-AGI reasoning benchmarks.
ARC-AGI
ARC-AGI is a visual grid puzzle benchmark created by François Chollet in 2019. It tests AI systems on novel problems that resist memorization and retrieval from large training data.
Arena-Hard
Arena-Hard is an automatic benchmark for instruction-tuned chat LLMs, released by LMSYS in April 2024, comparing each model head-to-head against a baseline using 500 challenging prompts.
BEIR
BEIR (Benchmarking-IR) is a benchmark for zero-shot evaluation of information retrieval models, covering 18 public datasets across nine tasks and multiple domains; it was created at TU Darmstadt and published at NeurIPS 2021.
BIG-bench
BIG-bench (Beyond the Imitation Game benchmark) is a crowdsourced benchmark of more than 200 tasks for large language models, introduced in June 2022 to test what models can do.
BIG-Bench Hard
BIG-Bench Hard (BBH) is a benchmark of 23 challenging multi-step reasoning tasks from BIG-Bench, introduced by Google researchers in 2022; with chain-of-thought prompting, Codex exceeded the average human rater on 17 of 23 tasks.
C-Eval
C-Eval is a Chinese-language benchmark for foundation models, released in May 2023 with 13,948 multiple-choice exam questions across 52 disciplines from middle school to professional level.
CMMLU
CMMLU, or Chinese Massive Multitask Language Understanding, is a Chinese-language knowledge benchmark for large language models with 11,528 multiple-choice questions across 67 subjects, released in June 2023.
CodeXGLUE
CodeXGLUE is a machine learning benchmark and open challenge for code intelligence, introduced in February 2021 by Microsoft researchers, with 10 tasks across 14 datasets and the CodeBLEU metric.
DS-1000
DS-1000 is a code generation benchmark of 1,000 data-science problems spanning seven Python libraries, built from real Stack Overflow questions and scored by executing generated code against test cases.
EQ-Bench
EQ-Bench is an independent benchmark suite measuring emotional intelligence and creative writing in large language models, first released in December 2023 as a 60-question test.
FACTS Grounding
FACTS Grounding is a benchmark and online leaderboard built by Google DeepMind and Google Research, announced in November 2024, that tests whether LLM answers stay fully grounded in a supplied document.
FrontierMath
FrontierMath is a benchmark of original, unpublished research-level mathematics problems created by Epoch AI, launched in November 2024, where leading models initially solved under 2%.
Gaokao Benchmark
The Gaokao benchmark (GAOKAO-Bench) is an evaluation suite testing large language models on questions from China's National College Entrance Examination, built by OpenLMLab at Fudan University in May 2023.
GLUE and SuperGLUE
GLUE and SuperGLUE, the General Language Understanding Evaluation benchmarks, are English natural language understanding benchmark suites, GLUE released in 2018 and SuperGLUE its harder 2019 successor.
GPQA
GPQA is a benchmark of graduate-level, Google-proof multiple-choice science questions written by PhD experts, published in 2023; its Diamond subset became the standard measure of frontier scientific reasoning.
GSM8K
GSM8K (Grade School Math 8K) is a benchmark of roughly 8,500 grade-school math word problems released by OpenAI in 2021, which became the standard test for chain-of-thought reasoning before saturating in the mid-2020s.
HealthBench
HealthBench is an open medical benchmark released by OpenAI in May 2025, measuring how language models handle realistic health conversations graded against rubrics written by 262 physicians.
HellaSwag
HellaSwag is a multiple-choice commonsense sentence-completion benchmark for language models, created at the University of Washington and Allen Institute for AI and published in 2019.
HumanEval
HumanEval is a benchmark of 164 hand-written Python programming problems, released by OpenAI in 2021 alongside Codex, measuring code generation by running outputs against unit tests.
HumanEval-X
HumanEval-X is a multilingual code-generation benchmark of 820 human-crafted programming problems in Python, C++, Java, JavaScript, and Go, built by the CodeGeeX team to extend HumanEval.
Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written academic questions launched by the Center for AI Safety and Scale AI in January 2025 to test frontier AI models.
IFEval (Instruction Following Evaluation)
IFEval, or Instruction Following Evaluation, is a Google Research benchmark released in November 2023 with 541 prompts checking whether large language models follow verifiable formatting and constraint instructions.
Language model benchmark
A language model benchmark is a standardized collection of tasks, test data, and scoring rules used to measure language model performance, spanning tests like MMLU and SWE-bench.