General · EdgepediaSeptember 17, 20267 min read

Humanity's Last Exam

Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written academic questions created by the Center for AI Safety and Scale AI to measure how close frontier large language models come to expert-level performance on closed-ended questions that cannot be answered by simple internet retrieval.¹ It was launched in January 2025 as a response to benchmark saturation, the tendency of frontier models to reach near-perfect scores on older evaluations such as MMLU, where state-of-the-art models now exceed 90% accuracy.² The first public results were strikingly low, with the best model at launch scoring 8.3%, but scores rose quickly, and independent audits later found that a substantial fraction of the questions themselves were defective.³^•⁴

Key fact	Detail
Creators and launch	Center for AI Safety and Scale AI; released January 2025, with the design study published in Nature on 28 January 2026²^•³
Size and coverage	2,500 questions across more than 100 subjects, grouped into eight high-level categories including mathematics, physics, chemistry, biology, medicine, computer science, humanities, and social sciences²
Question formats	24% multiple-choice with five or more options, the rest exact-match strings; about 14% require interpreting an image alongside text²
Contributors	Nearly 1,000 subject-matter experts, mostly professors, researchers, and graduate degree holders, from over 500 institutions in 50 countries, competing for a USD $500,000 prize pool²
Score at launch	OpenAI's o1 led the initial evaluation at 8.3%; human experts score around 90% in their own domains³
Calibration	Most models showed root mean square calibration errors above 70%, answering incorrectly with high stated confidence²
Audit verdicts	FutureHouse estimated 29 ± 3.7% of text-only chemistry and biology answers conflict with peer-reviewed literature; Epoch AI's 2026 review designated the original benchmark Flawed after finding errors in 22 of 48 sampled questions⁵^•⁴

Purpose and design

The benchmark addresses a measurement problem. By early 2025, large language models achieved more than 90% accuracy on popular benchmarks such as MMLU, which the HLE authors argued limited the ability to measure further progress at the frontier of human knowledge.² HLE was designed as what its creators called the final closed-ended academic benchmark of its kind: every question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval.¹

Submission criteria were strict. Questions had to be precise, unambiguous, solvable, and non-searchable, and each submission included the question text, answer specification, a detailed solution rationale, and the contributor's name and affiliation. Open-ended questions, subjective interpretations, and content related to weapons of mass destruction were prohibited.² To attract submissions, organizers offered a USD $500,000 prize pool, with $5,000 for each of the top 50 questions and $500 for each of the next 500, plus co-authorship on the paper for anyone with an accepted question.²

Construction and review

Question selection was filtered against the models of the day. Before submission, each question was tested against frontier LLMs, including GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and o1, with o1-mini and o1-preview added for text-only questions; exact-match questions had to stump all models, and multiple-choice questions had to stump all but one to allow for lucky guesses. More than 70,000 submission attempts produced roughly 13,000 questions that stumped the models, which then went to human expert review.²

Human review ran in two rounds with graduate-degree reviewers grading against standardized rubrics. The first round refined submissions, with each question receiving one to three reviews; the second round identified the questions approved for the final dataset. A private held-out set was kept apart from the public 2,500 questions to assess overfitting and gaming.² The initial arXiv preprint described 3,000 questions; the final released dataset contains 2,500.¹^•²

The organizers themselves measured reviewer disagreement. Two pre-release audits of 200 questions each yielded an estimated expert disagreement rate of 15.4% for the public set, which the authors described as in line with other machine learning benchmarks. A targeted peer review of a biology, chemistry, and health subset found disagreement of approximately 18%.²

Early results and score trajectory

At launch in January 2025, the researchers tested GPT-4o, o1, Gemini 1.5 Pro, Claude 3.5 Sonnet, and DeepSeek R1. OpenAI's o1 scored highest at 8.3%.³ The Nature paper reported low accuracy across all frontier models, and also found that models provided incorrect answers with high confidence rather than acknowledging uncertainty, with most models showing RMS calibration errors above 70%.² The paper also analyzed reasoning-token budgets: accuracy scaled log-linearly with more reasoning tokens, but the trend reversed after 2¹⁴ tokens, so a larger reasoning budget was not always optimal.²

Scores then climbed quickly. By July 2025, Grok 4 led with 26.9% accuracy alone and 44% with tools and internet access.⁵ As of 12 February 2026, the highest reported score was 48.4%, set by Google's Gemini 3 Deep Think.³ The Stanford HAI AI Index reported that frontier models gained 30 percentage points on HLE in a single year, on a benchmark built to be hard for AI and favorable to human experts.⁶

Audits and data-quality findings

FutureHouse audit. In July 2025, researchers at FutureHouse, working with experts in biology and chemistry, evaluated the answer-reasoning pairs of text-only chemistry and biology questions and found that 29 ± 3.7% (95% CI) had answers with directly conflicting evidence in peer-reviewed literature. They attributed the problem to the incentive structure: because inclusion required that frontier models fail the question, many adversarial "gotcha" questions were written whose answers the scientific literature contradicts. They released a validated subset, HLE Bio/Chem Gold.⁵ The HLE team responded with an independent three-expert review that found about 18% of a bio/chemistry subset problematic, and announced a rolling revision process.⁵

HLE-Verified. A February 2026 paper introduced HLE-Verified, a systematically verified revision of the benchmark. Its two-stage workflow certified 668 items as verified gold, revised 1,143 flawed but fixable items under preserved evaluation intent, and released 689 items as a documented uncertain set. Evaluating seven state-of-the-art models, the authors observed average absolute accuracy gains of 7 to 10 percentage points on HLE-Verified, rising to 30 to 40 percentage points on items whose original problem statement or reference answer was erroneous, indicating that part of measured performance on the original benchmark reflected annotation artifacts rather than genuine capability differences.⁷

Epoch AI review. In August 2026, Epoch AI, a research organization that tracks AI progress and reviews external benchmarks, audited a random sample of 48 HLE questions, six from each of the eight categories, using a model to surface candidate errors and confirming each by hand. It found 22 of 48 questions (46%) had substantial accuracy-altering errors: 12 were impossible to answer correctly as written, for reasons including missing assumptions, contradictory premises, multiple valid interpretations, or demands for objective answers to subjective questions, and 10 well-posed questions could produce false negatives, 5 of which could also produce false positives. Examples included a question asking for a 4-point discrete Fourier transform of an 8-entry sequence, and an abbreviation question accepting "1kin" for 1 Kings but grading the equally common "1kgs" incorrect. Epoch designated the benchmark Flawed, its verdict for benchmarks exceeding a 20% error threshold in the inspected sample.⁴

Criticism and limitations

The benchmark's name drew criticism in a Nature comment, which argued that "Humanity's Last Exam" could promote misunderstanding about the capacities of AI systems relative to humans.⁸ Commentators also noted that recalling esoteric graduate knowledge is a narrow slice of intelligence, and that a model could score well while failing at the sustained, tool-using, collaborative work that matters in practice.⁹ The creators themselves acknowledged the scope limit in the Nature paper: high accuracy on HLE would demonstrate expert-level performance on closed-ended, verifiable questions, but would not alone suggest autonomous research capabilities or artificial general intelligence, and the benchmark is intended to hasten a transition toward evaluations of more dynamic, open-ended capabilities.²

Because the public question set has been released, models can in principle be trained on it; the organizers maintain a private held-out set to detect overfitting and announced HLE-Rolling, a dynamic fork of the dataset regularly updated with community feedback and new questions.²

References

Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Evaluation, benchmarks, and leaderboards › Text, knowledge, and coding benchmarks

Initially written Sep 17, 2026 · Reviewed: Oct 15, 2026 · Edited: Oct 11, 2026; Oct 15, 2026 · Last review: Oct 15, 2026

Notice something wrong?