{ "id": "epe4150g48", "slug": "pairwise-comparison", "title": "Pairwise comparison", "updated": "2026-10-15", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics" }, { "id": "physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference" }, { "id": "physical.math.statistics.stats_statistical_inference.hypothesis_testing", "label": "Hypothesis testing", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference.hypothesis_testing" }, { "id": "physical.math.statistics.stats_statistical_inference.sequential_multiplicity.false_discovery_control", "label": "False discovery rate and error-rate control", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference.sequential_multiplicity.false_discovery_control" } ], "geo": [ { "id": "geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference.hypothesis_testing", "label": "Non-geographic · 1800 to 1945: Hypothesis testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference.hypothesis_testing", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1800", "label": "Non-geographic · 1800 to 1945", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800" }, { "id": "geo.nongeo.t1800.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical" }, { "id": "geo.nongeo.t1800.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math" }, { "id": "geo.nongeo.t1800.physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics" }, { "id": "geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference" }, { "id": "geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference.hypothesis_testing", "label": "Hypothesis testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference.hypothesis_testing" } ] } ], "excerpt": "A pairwise comparison is a statistical evaluation of one pair of treatments or items at a time, used to test differences between group means or to rank items from preferences.", "snippet": "A pairwise comparison is a statistical evaluation of one pair of treatments or items at a time, used to test differences between group means or to rank items from preferences.", "node": "physical.math.statistics.stats_statistical_inference.sequential_multiplicity.false_discovery_control", "markdown": "# Pairwise comparison\n\nA pairwise comparison is a statistical evaluation of one pair of treatments, groups, or items at a time, run across all pairs in a set either to test for differences between group means or to elicit preferences and rank items. The term covers two distinct practices: multiple-comparison testing after an experiment, which produces decisions, adjusted p-values, or simultaneous confidence intervals for differences such as \\( \\mu_{i} - \\mu_{j} \\), and paired-preference modeling, which produces strength estimates and rankings from win/loss data.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/) • [2](https://biostatisticsjdreeve.com/wp-content/uploads/2023/11/chapter-13-multiple-comparisons.pdf) • [3](https://ar5iv.labs.arxiv.org/html/1210.1016) The preference branch now underpins large-scale systems such as Chatbot Arena, which ranks language models from crowdsourced pairwise votes.[4](https://proceedings.mlr.press/v235/chiang24b.html)\n\n| Key fact | Value |\n|---|---|\n| Number of pairs among n items | \\( n(n-1)/2 \\): 5 groups → 10 pairs, 10 → 45, 20 → 190[5](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Pair-Wise_Multiple_Comparisons-Simulation.pdf) |\n| Type I error, 15 unadjusted tests (6 groups) | Over 50%[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/) |\n| Familywise error, independent tests | \\( \\alpha_{f} = 1 - (1 - \\alpha)^{C} \\); \\( \\alpha = 0.05 \\), \\( C = 10 \\) gives 0.401[5](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Pair-Wise_Multiple_Comparisons-Simulation.pdf) |\n| Tukey HSD simultaneous confidence coefficient | Exactly \\( 1 - \\alpha \\) with equal sample sizes; greater than \\( 1 - \\alpha \\) with unequal sizes (Tukey–Kramer)[6](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm) |\n| False discovery rate | Expected proportion of erroneous rejections among all rejections[7](https://home.cc.umanitoba.ca/~kesel/bookchap.pdf) |\n| Default parametric unplanned test | Tukey HSD; Tukey–Kramer variant when sample sizes differ[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/) |\n| Model-based win probability | Logistic link gives the Bradley–Terry model; normal link gives the Thurstone model[3](https://ar5iv.labs.arxiv.org/html/1210.1016) |\n\n## How it works\n\nWith \\( n \\) groups there are \\( n(n-1)/2 \\) pairs, so the comparison count grows quadratically: a 5-group design has 10 pairs, a 10-group design has 45, and a 20-group design has 190.[5](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Pair-Wise_Multiple_Comparisons-Simulation.pdf) Each individual test at the 0.05 level carries a 5% chance of a false positive, and these chances accumulate across the family. For independent tests the familywise error rate is \\( \\alpha_{f} = 1 - (1 - \\alpha)^{C} \\), which gives 0.401 for \\( \\alpha = 0.05 \\) and \\( C = 10 \\); for the correlated pairwise tests from one dataset this formula is an upper bound.[5](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Pair-Wise_Multiple_Comparisons-Simulation.pdf) Running all 15 pairwise comparisons among six groups without adjustment gives a Type I error probability over 50%.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/)\n\nTwo error rates define the design space. Familywise error rate (FWER) is the probability of at least one false rejection in the family; the false discovery rate (FDR), presented by Benjamini and Hochberg, is instead the expected proportion of erroneous rejections among total rejections.[7](https://home.cc.umanitoba.ca/~kesel/bookchap.pdf) Power also splits in two: any-pair power is the probability of detecting at least one true pair difference, and all-pairs power is the probability of detecting all of them, definitions recommended by Ramsey.[5](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Pair-Wise_Multiple_Comparisons-Simulation.pdf)\n\n## How it is done\n\nThe typical analysis tests all \\( a(a-1)/2 \\) comparisons of the form \\( \\mu_{i} - \\mu_{j} \\). For Tukey's HSD the test statistic is compared against a critical value from the Studentized range distribution \\( q_{(r,\\nu)} = w/s \\), where w is the range of r independent normal observations and s an independent estimate of their standard error with ν degrees of freedom, derived as the square root of the mean-square error; the decision rule rejects \\( H_{0}: \\mu_{i} = \\mu_{j} \\) when \\( |\\hat{D}_{ij}| \\ge \\frac{q_{(\\alpha,\\, a,\\, N-a)}}{\\sqrt{2}} \\cdot \\mathrm{se}(\\hat{D}_{ij}) \\).[8](https://math.montana.edu/jobo/st541/sec2c.pdf) • [6](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm) The resulting intervals are simultaneous: with equal sample sizes the overall probability that all intervals contain the true \\( \\mu_{i} - \\mu_{j} \\) is exactly \\( 1 - \\alpha \\), while the Tukey–Kramer intervals are conservative, with coverage at least \\( 1 - \\alpha \\), when sample sizes are unequal.[2](https://biostatisticsjdreeve.com/wp-content/uploads/2023/11/chapter-13-multiple-comparisons.pdf)\n\nCorrection choice follows the comparison family. Bonferroni rejects when the per-comparison p-value falls below \\( \\alpha/k \\), which for 5 groups means testing each pair at \\( \\alpha = 0.005 \\).[2](https://biostatisticsjdreeve.com/wp-content/uploads/2023/11/chapter-13-multiple-comparisons.pdf) Holm's multistage test adjusts the denominator depending on the number of null hypotheses remaining to be tested.[9](https://www.uvm.edu/~statdhtx/methods8/Supplements/OldChapter12.pdf) Assumptions mirror the independent-groups t test: normality, homogeneity of variance, and independent observations.[10](https://onlinestatbook.com/lms/tests_of_means/pairwise.html)\n\n## Origin\n\nOn the hypothesis-testing side, Tukey's all-pairs procedure is dated 1949 in one widely used review[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/); the two datings remain in use.[7](https://home.cc.umanitoba.ca/~kesel/bookchap.pdf) Scheffé's method for judging all contrasts appeared in [Biometrika](https://www.edgepedia.org/biometrika) in 1953,[11](https://doi.org/10.1093/biomet/40.1-2.87) Dunnett's procedure for comparing several treatments with a control appeared in the [Journal of the American Statistical Association](https://www.edgepedia.org/journal-of-the-american-statistical-association) in 1955,[12](https://doi.org/10.1080/01621459.1955.10501294) and Kramer's extension of multiple range tests to unequal replications in [Biometrics](https://www.edgepedia.org/biometrics) in 1956.[13](https://doi.org/10.2307/3001469) Hayter proved in 1984 that the Tukey–Kramer procedure is conservative.[14](https://doi.org/10.1214/aos/1176346392) Holm's sequentially rejective procedure followed in 1979[15](https://doi.org/10.2307/4615733) and Benjamini and Hochberg's FDR control in 1995.[16](https://doi.org/10.1111/j.2517-6161.1995.tb02031.x)\n\nOn the preference side, historical reviews place the Thurstone model of comparative judgment in [psychometrics](https://www.edgepedia.org/psychometrics), with Fechner regarded as the psychometric precursor.[17](https://pure.manchester.ac.uk/ws/files/50395219/Koczkodaj_LM_et_al_2015.pdf) The Bradley–Terry model for paired comparisons was published by [Ralph Allan Bradley](https://www.edgepedia.org/ralph-a-bradley) and [Milton E. Terry](https://www.edgepedia.org/milton-e-terry) in Biometrika in 1952,[18](https://doi.org/10.2307/2334029) and Rao and Kupper generalized it to handle ties in 1967.[19](https://doi.org/10.1080/01621459.1967.10482901) Chiang and colleagues introduced Chatbot Arena, an open platform for evaluating LLMs by human preference, in 2024.[4](https://proceedings.mlr.press/v235/chiang24b.html)\n\n## Variants\n\n**Hypothesis-testing procedures** differ in scope and stringency. Tukey HSD covers all pairs and, when only pairwise comparisons are made, produces the narrowest simultaneous confidence intervals.[20](https://online.stat.psu.edu/stat502/book/export/html/841) Fisher's LSD is a protected t-test run only after a significant ANOVA F; it has high power but poor control of the experimentwise error rate and is generally not recommended for unplanned comparisons.[8](https://math.montana.edu/jobo/st541/sec2c.pdf) Scheffé's method covers all contrasts, protects against data snooping, and is the most conservative choice, with the lowest power.[20](https://online.stat.psu.edu/stat502/book/export/html/841) • [21](https://pmc.ncbi.nlm.nih.gov/articles/PMC11637079/) Dunnett's procedure compares \\( a-1 \\) treatments against one control, gaining power by testing fewer comparisons.[8](https://math.montana.edu/jobo/st541/sec2c.pdf) • [20](https://online.stat.psu.edu/stat502/book/export/html/841) When variances are unequal, only the four procedures designed for that setting, Dunnett's C, Dunnett's T3, Games–Howell, and Tamhane's T2, adequately controlled Type I error in simulations, with Games–Howell slightly highest in power.[22](https://journals.sagepub.com/doi/10.1177/2515245918808784) Shaffer's modified sequentially rejective procedure (1986) refines the step-down family.[23](https://doi.org/10.1080/01621459.1986.10478341)\n\n**Model-based approaches** estimate item strengths instead of testing differences. A paired-comparison model with win probability \\( P(i \\succ j) = F(\\lambda_{i} - \\lambda_{j}) \\) becomes the Thurstone model when F is the normal cumulative distribution function and the [Bradley–Terry model](https://www.edgepedia.org/bradley-terry-model) when F is logistic.[3](https://ar5iv.labs.arxiv.org/html/1210.1016) The Bradley–Terry model is the basis for the [Elo rating system](https://www.edgepedia.org/elo-rating-system) used in chess and other repeated competitions.[24](https://betanalpha.github.io/assets/chapters_pdf/pairwise_comparison_modeling.pdf)\n\n## Applications\n\nMultiple-comparison tests are workhorse tools in biology and psychology: a survey of the environmental and biological literature over the past 20 years found Tukey HSD used in 30.04% of reports, while Games-Howell (1.13%), Holm-Bonferroni (1.25%), and Scheffé (2.25%) were least used.[21](https://pmc.ncbi.nlm.nih.gov/articles/PMC11637079/) Paired-comparison models serve sensory and audio evaluation, [journal ranking](https://www.edgepedia.org/journal-ranking), genetics, and risk assessment.[3](https://ar5iv.labs.arxiv.org/html/1210.1016) The largest recent application is LLM evaluation: Chatbot Arena collects crowdsourced pairwise human preferences; by 2026 the platform (now rebranded to 'Arena', at arena.ai) had amassed millions of votes, with a 2026 copy of the Text (Overall, style-controlled) leaderboard reporting 7,874,713 total votes across 393 models.[4](https://proceedings.mlr.press/v235/chiang24b.html)\n\n## Limitations and alternatives\n\n**Error inflation and power loss.** Uncorrected pairwise testing inflates false positives, and even nominally protected procedures can fail: under non-complete-null configurations the Fisher LSD familywise error reaches 0.1222 for 4 groups and 0.9044 for 8 groups at \\( \\alpha = 0.05 \\).[7](https://home.cc.umanitoba.ca/~kesel/bookchap.pdf) Tukey is more powerful than Bonferroni or Šidák when all pairs are of interest, but its critical value depends on the number of treatments a rather than the number of comparisons C, so it loses power when only a few pairs matter.[8](https://math.montana.edu/jobo/st541/sec2c.pdf)\n\n**Intransitivity and alternatives.** Intransitive decisions, in which A beats B, B beats C, but C beats A, are extremely common with conventional procedures; an AIC-based model-testing approach has been proposed to avoid them with better all-pairs power than Tukey HSD.[7](https://home.cc.umanitoba.ca/~kesel/bookchap.pdf) Non-parametric options exist for rank data: the Mann-Whitney-Wilcoxon U test for planned comparisons, and the Dunn, Nemenyi, and Steel-Dwass procedures for unplanned ones; the parametric Games–Howell procedure, used under unequal variances, is not a rank-based alternative.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/)\n\n## References\n\n1. [Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test (Midway et al., 2020, PeerJ)](https://pmc.ncbi.nlm.nih.gov/articles/PMC7720730/)\n2. [Chapter 13: Multiple Comparisons (biostatistics text, J.D. Reeve)](https://biostatisticsjdreeve.com/wp-content/uploads/2023/11/chapter-13-multiple-comparisons.pdf)\n3. [Models for Paired Comparison Data: A Review with Emphasis on Dependent Data (Cattelan)](https://ar5iv.labs.arxiv.org/html/1210.1016)\n4. [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024)](https://proceedings.mlr.press/v235/chiang24b.html)\n5. [Pair-Wise Multiple Comparisons (Simulation), PASS/NCSS documentation](https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Pair-Wise_Multiple_Comparisons-Simulation.pdf)\n6. [7.4.7.1. Tukey's method, NIST/SEMATECH e-Handbook of Statistical Methods](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm)\n7. [Pairwise Multiple Comparison Test Procedures (Keselman et al., book chapter)](https://home.cc.umanitoba.ca/~kesel/bookchap.pdf)\n8. [ST 541 course notes: Multiple Comparison Procedures (Montana State University)](https://math.montana.edu/jobo/st541/sec2c.pdf)\n9. [Chapter 12: Multiple Comparisons Among Treatment Means (Howell, textbook chapter)](https://www.uvm.edu/~statdhtx/methods8/Supplements/OldChapter12.pdf)\n10. [All Pairwise Comparisons Among Means (Lane, OnlineStatBook)](https://onlinestatbook.com/lms/tests_of_means/pairwise.html)\n11. [HENRY SCHEFFÉ (1953). A METHOD FOR JUDGING ALL CONTRASTS IN THE ANALYSIS OF VARIANCE *. Biometrika.](https://doi.org/10.1093/biomet/40.1-2.87)\n12. [Charles W. Dunnett (1955). A Multiple Comparison Procedure for Comparing Several Treatments with a Control. Journal of the American Statistical Association.](https://doi.org/10.1080/01621459.1955.10501294)\n13. [Clyde Young Kramer (1956). Extension of Multiple Range Tests to Group Means with Unequal Numbers of Replications. Biometrics.](https://doi.org/10.2307/3001469)\n14. [Anthony J. Hayter (1984). A Proof of the Conjecture that the Tukey-Kramer Multiple Comparisons Procedure is Conservative. The Annals of Statistics.](https://doi.org/10.1214/aos/1176346392)\n15. [Sture Holm (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics.](https://doi.org/10.2307/4615733)\n16. [Yoav Benjamini, Yosef Hochberg (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B (Statistical Methodology).](https://doi.org/10.1111/j.2517-6161.1995.tb02031.x)\n17. [Important Facts and Observations about Pairwise Comparisons (Koczkodaj et al., 2015)](https://pure.manchester.ac.uk/ws/files/50395219/Koczkodaj_LM_et_al_2015.pdf)\n18. [Ralph Allan Bradley, Milton E. Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika.](https://doi.org/10.2307/2334029)\n19. [P. V. Rao, L. L. Kupper (1967). Ties in Paired-Comparison Experiments: A Generalization of the Bradley-Terry Model. Journal of the American Statistical Association.](https://doi.org/10.1080/01621459.1967.10482901)\n20. [2.4 - Other Pairwise Mean Comparison Methods (STAT 502, Penn State)](https://online.stat.psu.edu/stat502/book/export/html/841)\n21. [On the use of post-hoc tests in environmental and biological sciences: A critical review (2024)](https://pmc.ncbi.nlm.nih.gov/articles/PMC11637079/)\n22. [An Updated Recommendation for Multiple Comparisons (2019, AMPPS)](https://journals.sagepub.com/doi/10.1177/2515245918808784)\n23. [Juliet Popper Shaffer (1986). Modified Sequentially Rejective Multiple Test Procedures. Journal of the American Statistical Association.](https://doi.org/10.1080/01621459.1986.10478341)\n24. [Pairwise Comparison Modeling (Betancourt)](https://betanalpha.github.io/assets/chapters_pdf/pairwise_comparison_modeling.pdf)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Statistical inference, estimation, sampling, and testing › Hypothesis testing › False discovery rate and error-rate control*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://www.uvm.edu/~statdhtx/methods8/Supplements/OldChapter12.pdf" ], "url": "https://www.edgepedia.org/pairwise-comparison", "markdown_url": "https://www.edgepedia.org/pairwise-comparison.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Pairwise comparison\", Edgepedia (EdgeChat), https://www.edgepedia.org/pairwise-comparison. Edgepedia Community License 1.0.", "credit_md": "\"[Pairwise comparison](https://www.edgepedia.org/pairwise-comparison)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/pairwise-comparison](https://www.edgepedia.org/pairwise-comparison). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Pairwise comparison\", Edgepedia (EdgeChat), https://www.edgepedia.org/pairwise-comparison. Edgepedia Community License 1.0.", "speakable": "A pairwise comparison is a statistical evaluation of one pair of treatments or items at a time, used to test differences between group means or to rank items from preferences." }