# MT-Bench
MT-Bench is a benchmark of 80 two-turn conversation questions, built in June 2023 by researchers at LMSYS (Large Model Systems Organization) to measure a large language model's multi-turn conversation and instruction-following ability, scored by using another LLM as the judge. It appeared in the paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" by Lianmin Zheng and colleagues, first posted to arXiv in June 2023 and published in the NeurIPS 2023 Datasets and Benchmarks track.[1](https://arxiv.org/html/2306.05685v1) • [2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)
| Fact | Detail |
|---|---|
| Design | 80 questions, 8 categories, 10 questions per category, each exactly two turns[1](https://arxiv.org/html/2306.05685v1) |
| Categories | Writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, humanities/social-science knowledge[1](https://arxiv.org/html/2306.05685v1) |
| Default scoring | Single-answer grading: GPT-4 scores each turn 1–10; score is the average over all turns[3](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm%5Fjudge/README.md) |
| Headline 2023 score (NeurIPS version) | GPT-4 8.99, GPT-3.5 7.94, Vicuna-13B 6.39[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf) |
| Judge-human agreement | 85% without ties, above the 81% human-human baseline; about 66% once ties and position-inconsistent votes are included[1](https://arxiv.org/html/2306.05685v1) • [4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) |
| Human annotations released | 3.3K judgments covering 6 models on the 80 questions[3](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm%5Fjudge/README.md) |
| Status, September 2026 | Saturated above 9.0 at the frontier; the tracked leaderboard holds 13 self-reported, unverified entries[5](https://llm-stats.com/benchmarks/mt-bench) • [6](https://futureagi.com/glossary/mt-bench-conversation-benchmark/) |
## What MT-Bench is
The benchmark tests multi-turn conversation and instruction-following, not stored knowledge. Its authors identified 8 common categories of user prompts and wrote 10 manually designed two-turn questions for each: writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge II (humanities and social science). Questions were chosen to be challenging enough to separate models rather than to cover a curriculum.[1](https://arxiv.org/html/2306.05685v1) • [2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)
The second turn is the point of the design. Each scenario is an initial prompt plus one predefined follow-up, so a model must carry context forward and revise or extend a prior answer. Single-turn benchmarks cannot see whether a model handles that follow-through; MT-Bench was built so they would have to.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) The sources state the 80-question design but not why that size was chosen rather than something larger.
The same study introduced Chatbot Arena, which collects crowdsourced pairwise human votes, so LLM-judge scores could be checked against human preferences on the same questions.[1](https://arxiv.org/html/2306.05685v1)
## How the scoring works
**LLM-as-a-judge** replaces human raters with a strong model. The LMSYS team recommends single-answer grading as the default mode: GPT-4, the default judge, reads a model's answer and assigns a score on a scale of 10 for each turn, and the benchmark score is the average over all turns. The FastChat llm_judge package also supports two win-rate modes: pairwise comparison against a baseline model (default gpt-3.5-turbo) and all-pairs comparison.[3](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm%5Fjudge/README.md) For math and reasoning questions, a reference-guided variant supplies reference answers to the judge.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark))
The two modes trade off differently. [Pairwise comparison](https://www.edgepedia.org/pairwise-comparison) scales poorly as models multiply, since the number of pairs grows quadratically; single-answer grading may miss subtle differences between specific pairs and fluctuates more when the judge model changes.[1](https://arxiv.org/html/2306.05685v1) To let others calibrate judges, LMSYS released 3.3K human annotations of responses from 6 models on the 80 questions.[3](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm%5Fjudge/README.md)
## Published results and timeline
The published numbers themselves illustrate a reproducibility problem. The June 2023 arXiv v1 reported GPT-4 8.9, GPT-3.5 8.7, Vicuna-13B 7.3, Vicuna-7B 7.1, Alpaca-13B 6.1, LLaMA-13B 4.1, and LLaMA-7B 4.3.[1](https://arxiv.org/html/2306.05685v1) The peer-reviewed NeurIPS version's Table 7 gives materially different values for the same models: GPT-4 8.99, GPT-3.5 7.94, Vicuna-13B 6.39, Vicuna-7B 6.00, Alpaca-13B 4.53, LLaMA-13B 2.61, and LLaMA-7B 2.74. The gap between versions shows how sensitive scores are to the judge model and its version.[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)
Category-level results carried their own lesson. In math and coding, GPT-3.5 and GPT-4 had similar overall win-rates because both failed some hard questions, yet GPT-4 was significantly better in direct pairwise comparison or single-answer grading, a reminder that aggregate scores can mask category-level differences.[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)
By September 2026 the benchmark's leaderboard is a shadow of its former role. LLM Stats tracks 13 models on MT-Bench, all self-reported and none independently verified, led by Hermes 3 70B from Nous Research at 8.990, with Hermes 3 405B second at 8.930. The same table mixes scoring scales, with some entries near 0.9 and one at 0.090, an example of inconsistent score reporting on the aging benchmark.[5](https://llm-stats.com/benchmarks/mt-bench)
## By the numbers: judge-human agreement
The paper's headline claim is that GPT-4 as judge matched human evaluations at an agreement rate exceeding 80%, based on 3K controlled expert votes and 3K crowdsourced human votes, reaching the same level as human-human agreement.[1](https://arxiv.org/html/2306.05685v1) The precise figures: under setup S2, excluding ties, GPT-4 agreed with humans 85% of the time, higher than the 81% agreement among humans themselves.[1](https://arxiv.org/html/2306.05685v1) When humans disagreed with GPT-4, they deemed its judgment reasonable in 75% of cases and changed their own choice in 34% of cases.[1](https://arxiv.org/html/2306.05685v1)
The 85% figure has conditions. A reference summary reports that agreement dropped to roughly 66% once ties and position-inconsistent votes were included.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) Agreement also depends on how far apart the compared models are: it rises from 70% to nearly 100% as the win-rate gap between the pair grows, so the headline number is best read as an average over mixed pairs, not a constant.[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf) The successor benchmark MT-Bench-101, using detailed scoring guidelines and a minimum-score metric, measured 87% GPT-4-judge agreement with human experts, above its 80% internal human-expert agreement.[7](https://aclanthology.org/2024.acl-long.401.pdf)
## Criticisms and limitations
**Judge biases.** The original paper examined position bias, verbosity bias, self-enhancement bias, and limited reasoning ability, finding some minor or mitigable. In pairwise judging, only GPT-4 gave position-consistent judgments in more than 60% of cases; most LLM judges strongly favor the first answer shown.[1](https://arxiv.org/html/2306.05685v1) • [2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf) Position bias in pairwise mode can be mitigated by swapping answer order and counting only consistent wins, but single-answer grading remains exposed to verbosity and self-enhancement bias.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) Self-bias, a judge preferring its own family's answers (GPT-4 judge preferring GPT-4 answers), was documented by independent researchers; the MT-Bench-101 authors found the problem minor in their setting, with a Qwen-72B judge producing rankings consistent with a GPT-4 judge.[7](https://aclanthology.org/2024.acl-long.401.pdf)
**Small, static, narrow.** With 80 questions and 160 turns, the benchmark is small; strong models cluster near the top, and a static question set is vulnerable to training-data contamination.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) Its 80 fixed questions also say little about domain grounding, tone, or policy-specific refusals.[6](https://futureagi.com/glossary/mt-bench-conversation-benchmark/) The judge itself is imperfect: it cannot always catch factual errors or hallucinations, and in the original paper it was sometimes misled by supplied answers even on problems it could solve on its own.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) The focus is helpfulness, not safety or honesty.[4](https://systems-analysis.ru/eng/MT-Bench_(benchmark)) On whether MT-Bench questions demonstrably appeared in specific training sets, the sources document only the generic contamination risk, not confirmed cases.
A further caution comes from the paper itself: fine-tuning Vicuna-7B on a small, high-quality conversation dataset (4.8M tokens, 3K conversations) quickly taught a GPT-4-preferred style without significantly improving MMLU, showing that judge-approved style can be learned separately from capability.[2](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)
## What changed after 2023: saturation and successors
By 2026, MT-Bench is saturated at the frontier: top closed-source models cluster above 9.0, and the remaining headroom is mostly judge-model noise. It remains more useful as a sanity check on smaller open-weight models, where scores in the 7–8.5 range still discriminate.[6](https://futureagi.com/glossary/mt-bench-conversation-benchmark/) No source pins a specific date or threshold at which models hit the ceiling; saturation above 9.0 is documented only as a 2026 state.
The direct successor for multi-turn evaluation is MT-Bench-101, posted to arXiv on 2024-02-22, accepted to ACL 2024 on 2024-05-15, and released with code and dataset on 2024-05-28.[8](https://github.com/mtbench101/mt-bench-101) It uses a three-tier ability taxonomy (perceptivity, adaptability, interactivity) spanning 13 tasks with 4,208 turns across 1,388 multi-turn dialogues, and identifies adaptability and interactivity as the key deficiencies of existing LLMs. Across 21 evaluated models, GPT-4 led with an average score of 8.86 and Yi-34B was second at 8.10, with mathematical reasoning the hardest task.[7](https://aclanthology.org/2024.acl-long.401.pdf) Its authors also found that neither common alignment techniques such as RLHF nor chat-specific designs significantly improved multi-turn abilities.[7](https://aclanthology.org/2024.acl-long.401.pdf)
Current guidance pairs MT-Bench with newer benchmarks such as [AgentBench](https://www.edgepedia.org/agentbench) for tool use and GAIA for multi-step reasoning, rather than relying on it alone.[6](https://futureagi.com/glossary/mt-bench-conversation-benchmark/)
## References
1. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., June 2023)](https://arxiv.org/html/2306.05685v1)
2. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023 Datasets and Benchmarks version)](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)
3. [FastChat llm_judge README (LMSYS)](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm%5Fjudge/README.md)
4. [MT-Bench (benchmark) — systems-analysis.ru reference entry](https://systems-analysis.ru/eng/MT-Bench_(benchmark))
5. [MT-Bench Leaderboard — LLM Stats](https://llm-stats.com/benchmarks/mt-bench)
6. [What Is MT-Bench? FutureAGI Guide (2026)](https://futureagi.com/glossary/mt-bench-conversation-benchmark/)
7. [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues (ACL 2024)](https://aclanthology.org/2024.acl-long.401.pdf)
8. [mtbench101/mt-bench-101 (GitHub repository)](https://github.com/mtbench101/mt-bench-101)
---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Evaluation, benchmarks, and leaderboards › Text, knowledge, and coding benchmarks*
*Initially written Sep 17, 2026 · Edited: Oct 11, 2026; Oct 15, 2026*
*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*
License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license