# AlpacaEval
AlpacaEval is an automatic, LLM-judged benchmark, and leaderboard for instruction-following chat models, built by researchers at Stanford and released on GitHub in May 2023. It measures how often a model's answers are preferred by a GPT-4-based judge over the answers of a fixed reference model on a set of 805 instructions, replacing costly human evaluation with a cheap, reproducible proxy.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) • [2](https://arxiv.org/html/2404.04475v1) Its best-known contribution is the length-controlled win rate introduced with AlpacaEval 2.0 in 2024, a statistical correction for the tendency of LLM judges to prefer longer answers.[2](https://arxiv.org/html/2404.04475v1)
| Key fact | Detail |
|---|---|
| Creators | Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto (Stanford); released 2023[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) |
| Test set | 805 instructions drawn from self-instruct, open-assistant, vicuna, koala, and hh-rlhf[2](https://arxiv.org/html/2404.04475v1) • [1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) |
| Versions | AlpacaEval 1.0 (text-davinci-003 reference); AlpacaEval 2.0 (gpt4_turbo reference, length-controlled win rate)[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) |
| Judge | GPT-4-based auto-annotators; AlpacaEval 2.0 uses weighted_alpaca_eval_gpt4_turbo with logprob-based continuous preferences[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) |
| Cost and speed | Under $10 of OpenAI credits and under 3 minutes per run (maintainer-reported)[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) |
| Agreement with humans | 0.98 Spearman correlation with LMSYS Chatbot Arena after length control[2](https://arxiv.org/html/2404.04475v1) |
| Status | Last official leaderboard snapshot dated December 27, 2024; no 2026 frontier models covered[3](https://klu.ai/glossary/alpaca-eval) |
## What AlpacaEval is
AlpacaEval began as a development metric for the Alpaca chatbot (Taori et al., 2023) and the AlpacaFarm evaluation simulator (Dubois et al., 2023), two Stanford projects that needed a fast way to compare fine-tuned models during training.[2](https://arxiv.org/html/2404.04475v1) It operates on a fixed set of 805 instructions chosen to be representative of user interactions on the Alpaca web demo.[2](https://arxiv.org/html/2404.04475v1) When other labs began submitting results, the project turned into a public leaderboard, and the paper's authors note that this subsequent use revealed the uncontrolled biases could be significantly gamed in ways human biases could not.[2](https://arxiv.org/html/2404.04475v1)
The maintainers are explicit about scope: AlpacaEval measures instruction-following ability only, not the harm a model could cause, and it does not evaluate safety. They caution that the small gap it shows between ChatGPT and the best open-source models should not be read as deployment readiness.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md)
## How the evaluation works
The metric is a pairwise win rate. Each model's output for an instruction is paired with the reference model's output on the same instruction, and a GPT-4-based automatic annotator is asked which output it prefers. The win rate is the fraction of instructions on which the model's output wins; identical outputs earn each side half a preference. The order of the two outputs is randomized to avoid position bias.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) The reference model is text-davinci-003 for AlpacaEval 1.0 and gpt4_turbo for AlpacaEval 2.0, the upgrade made to keep the benchmark challenging as models improved.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md)
AlpacaEval 2.0's headline metric is the length-controlled win rate: a regression adjustment that re-weights the judge's preferences as if the two outputs had no difference in length, so that a model cannot raise its score simply by writing more.[4](https://atlas.kevinhu.io/benchmarks/alpacaeval-2-lc-winrate) The 2.0 annotator, weighted_alpaca_eval_gpt4_turbo, also uses logprob-based continuous preferences rather than a binary choice, so a barely-preferred output counts for less than a decisively preferred one.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md)
## By the numbers
The motivation for length control is visible in one number: varying only the verbosity instruction in the prompt moved the raw win rate of the baseline model gpt4_1106_preview from 22.9% to 64.3%, a swing of more than 40 percentage points produced by no change in answer quality.[2](https://arxiv.org/html/2404.04475v1) The length-controlled win rate narrows that fluctuation to 41.9% to 51.6% and cuts the normalized standard deviation across three verbosity prompts from 25% to 10%.[2](https://arxiv.org/html/2404.04475v1)
Length control also improved external validity. The authors report that it raised AlpacaEval's Spearman correlation with [LMSYS Chatbot Arena](https://www.edgepedia.org/lmsys-chatbot-arena) from 0.94 to 0.98, which they describe as the highest known correlation with Chatbot Arena among benchmarks evaluating at least 25 models. (The GitHub README states the same post-correction figure of 0.98 but gives 0.93 as the pre-correction value; the paper's 0.94 is used here.)[2](https://arxiv.org/html/2404.04475v1) • [1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md)
Gaming remains possible even after the correction. In a white-box adversarial attack that post-processed GPT-4's outputs by truncating most of them, the win rate rose from 3.7 (AlpacaEval 2.0) to 25.9 (length-controlled, without regularization); the regularization term included in the method reduces the gamed score to 12.2.[2](https://arxiv.org/html/2404.04475v1)
Later third-party snapshots show the benchmark still being applied to 2024–2025 models: llm-stats.com lists [IBM Granite](https://www.edgepedia.org/ibm-granite) 3.3 8B (Base and Instruct) tied at 0.627, DeepSeek-V2.5 (236B parameters) at 0.505, and IBM Granite 4.0 Tiny Preview at 0.352.[5](https://llm-stats.com/benchmarks/alpacaeval-2.0)
## Comparison with other leaderboards
The AlpacaEval authors treat LMSYS Chatbot Arena, where users vote on anonymous model outputs, as a silver standard: the largest and most ecologically valid human evaluation process available.[2](https://arxiv.org/html/2404.04475v1) Against that standard, length-controlled AlpacaEval correlates at 0.98 Spearman.[2](https://arxiv.org/html/2404.04475v1)
The trade-off is cost and control. Chatbot Arena reflects real user preferences but requires continuous human traffic; AlpacaEval produces a full leaderboard run in under 3 minutes for under $10 of OpenAI credits, so any lab can reproduce it, but its judge inherits the biases of a commercial LLM rather than of the human population.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) • [2](https://arxiv.org/html/2404.04475v1)
## Criticisms and gaming
**Judge bias is documented by the project itself.** The maintainers report that automatic annotators prefer longer outputs and outputs containing lists (0.68/0.69 preference rates for the alpaca_eval_gpt4 annotator) and that automatic evaluators weight style over factuality. The paper's crowd-worker validation found the same pattern in humans: annotators favor style such as length and list presence over factuality, and the authors note that 16 crowd-workers' preferences are not representative of all humans.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) • [2](https://arxiv.org/html/2404.04475v1)
**Results depend on the exact judge version.** In April 2024, a user reported that judging claude-3-opus-20240229 with gpt-4-turbo-2024-04-09 produced a score of 47.38, versus 40.39 on the leaderboard, and 41.09 with gpt-4-0125-preview as judge. A maintainer confirmed the annotator had been switched to gpt-4-turbo after limited testing on three models; broader testing on 15 models showed large differences for a few of them, and a follow-up issue reverted the annotator to the preview version and re-annotated recently added models. The issue also recommends pinning specific version numbers, because symbolic names like `gpt-turbo` and `gpt-preview` point to frequently updated models and make leaderboard results irreproducible.[6](https://github.com/tatsu-lab/alpaca_eval/issues/282)
**Verification is partial.** A verified result means a core maintainer decoded the model's outputs and performed the evaluation; the maintainers state they lack the resources to verify all models, so only top-5 leaderboard entries are verified and other results are unverified.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md)
**The test set has known limits.** The maintainers caution that the 805 instructions, drawn from [self-instruct](https://www.edgepedia.org/self-instruct), open-assistant, vicuna, koala, and hh-rlhf, might not represent real usage, and that the mix likely makes closed models appear more similar to open models than they are.[1](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md) The evidence available does not include an independent contamination study of the test set, so whether training-data contamination affects scores is not settled by these sources.
## References
1. [tatsu-lab/alpaca_eval README (GitHub)](https://github.com/tatsu-lab/alpaca_eval/blob/main/README.md)
2. [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (arXiv, April 2024)](https://arxiv.org/html/2404.04475v1)
3. [AlpacaEval — Klu](https://klu.ai/glossary/alpaca-eval)
4. [AlpacaEval 2.0 (length-controlled win rate) — Benchmark Atlas](https://atlas.kevinhu.io/benchmarks/alpacaeval-2-lc-winrate)
5. [AlpacaEval 2.0 Leaderboard summary (llm-stats.com)](https://llm-stats.com/benchmarks/alpacaeval-2.0)
6. [Confusion in Model Evaluation Results Due to GPT Updates · Issue #282 (GitHub)](https://github.com/tatsu-lab/alpaca_eval/issues/282)
---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Evaluation, benchmarks, and leaderboards › Text, knowledge, and coding benchmarks*
*Initially written Sep 17, 2026 · Edited: Oct 11, 2026; Oct 15, 2026; Oct 21, 2026*
*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*
License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license