{ "id": "ep9d712w22", "slug": "mantel-test", "title": "Mantel test", "updated": "2026-10-15", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics" }, { "id": "physical.math.statistics.multivariate-analysis", "label": "Multivariate association and dimension reduction", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.multivariate-analysis" }, { "id": "physical.math.statistics.multivariate-analysis.alpha-a-to-me", "label": "Titles A to Me", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.multivariate-analysis.alpha-a-to-me" } ], "geo": [ { "id": "geo.nongeo.t1946.physical.math.statistics.multivariate-analysis", "label": "Non-geographic · 1946 to 2000: Multivariate association and dimension reduction", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.statistics.multivariate-analysis", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical" }, { "id": "geo.nongeo.t1946.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math" }, { "id": "geo.nongeo.t1946.physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.statistics" }, { "id": "geo.nongeo.t1946.physical.math.statistics.multivariate-analysis", "label": "Multivariate association and dimension reduction", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.statistics.multivariate-analysis" } ] } ], "excerpt": "The Mantel test is a permutation-based statistical test that assesses the association between two distance matrices for the same objects, such as geographic and genetic distances.", "snippet": "The Mantel test is a permutation-based statistical test that assesses the association between two distance matrices for the same objects, such as geographic and genetic distances.", "node": "physical.math.statistics.multivariate-analysis.alpha-a-to-me", "markdown": "# Mantel test\n\nThe Mantel test is a permutation-based statistical test that assesses the association between two square matrices whose entries are distances, dissimilarities, or similarities among the same n objects. It has been applied in different fields of the life sciences.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/) In its simplest form it compares a matrix of geographic distances to a matrix of genetic or other dissimilarities among the same objects.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) Because the pairwise entries of a distance matrix are mutually dependent, the usual significance tests are invalid, and the test instead builds its null distribution by permuting whole objects.[3](https://vegandevs.github.io/vegan/reference/mantel.html)\n\n| Key fact | Detail |\n|---|---|\n| Statistic | The Mantel correlation is the Pearson correlation between the corresponding upper-triangular entries of two distance matrices; it ranges from −1 to +1.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) • [4](https://uw.pressbooks.pub/appliedmultivariatestatistics/chapter/mantel-test/) |\n| Null hypothesis | Distances among objects in matrix A are linearly independent of distances among the same objects in matrix B.[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf) |\n| Randomization | One matrix is held rigid while the rows and corresponding columns of the other are permuted together, preserving matrix symmetry.[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf) |\n| Permutation count | 999 permutations are common in worked examples, but probability estimates from 500–2000 permutations can carry 5–6% error; 10,000 or more are recommended for stable p-values.[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/) • [8](https://doi.org/10.1139/z89-108) |\n| Main variants | Partial Mantel test, Mantel correlogram, multiple regression on distance matrices (MRM), and CADM.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) |\n| Known failure mode | Under spatial autocorrelation in both response and explanatory variables, Type I error rates can reach 25–55% at a nominal 5% level.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018) |\n| Software | The vegan package computes the Mantel and partial Mantel tests; ecodist provides the partial Mantel r and MRM.[3](https://vegandevs.github.io/vegan/reference/mantel.html) • [10](https://search.r-project.org/CRAN/refmans/ecodist/html/mantel.html) |\n\n## How it works\n\nThe Mantel test is a test for association between two \\( n \\times n \\) matrices whose entries are distances or similarities between all pairs of objects.[11](https://people.clas.ufl.edu/jlichstein/files/Lichstein_2007_Plant_Ecol.pdf) His original statistic was the cross-product of the distances in the two matrices, that is, the sum over all pairs of the product of the two distance values.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) • [7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/) Modern programs instead compute the Mantel correlation, the cross-product between standardized distances divided by \\( (d - 1) \\), where \\( d \\) is the number of distances in the upper-triangular portion of each matrix; this is simply the Pearson correlation between the corresponding entries, bounded between −1 and +1, and the transformation has no effect on the permutation-test probability.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) • [7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)\n\nThe null hypothesis is that the distances among objects in matrix A are linearly independent of the distances among the same objects in matrix B.[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf) For symmetric matrices only the upper (or lower) triangular entries are used; for non-symmetric matrices both portions are included, and the main diagonal need not be included.[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)\n\nThe reason permutation is required is structural: an \\( n \\times n \\) distance matrix contains \\( n(n - 1)/2 \\) entries for only \\( n \\) observations, so the entries are mutually dependent and the usual significance tests are invalid.[3](https://vegandevs.github.io/vegan/reference/mantel.html) • [6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf) Under the null hypothesis, the objects, not the distances, are the permutable units.[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf) Randomization therefore permutes the n objects of one matrix, which is equivalent to permuting its rows and corresponding columns together: if rows one and four are swapped, columns one and four are also exchanged, so the matrix stays symmetric.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) • [12](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)\n\n## How it is done\n\nThe procedure has four steps.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)\n\n1. Build the two \\( n \\times n \\) distance matrices from the same set of objects.\n2. Compute the scalar product (or equivalently the Mantel correlation) between the corresponding entries.\n3. For a large number of iterations, draw a [random permutation](https://www.edgepedia.org/random-permutation) of {1, …, n} uniformly, apply it to the rows and corresponding columns of one matrix, and recompute the statistic; one matrix is held rigid while the other is permuted.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018) • [6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)\n4. Derive the p-value by comparing the observed statistic with the permuted values, typically from the upper tail for a one-sided test of positive correlation.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018) • [3](https://vegandevs.github.io/vegan/reference/mantel.html)\n\nWith 999 randomizations, if none of the randomized values exceeds the observed one, the p-value is reported as \\( 1/1000 \\), since the observed value is conservatively added to both numerator and denominator.[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/) More generally the permutation p-value is \\( p = (N + 1)/(n + 1) \\), where \\( n \\) is the number of permutations, so the minimum achievable p-value is \\( 1/(n + 1) \\).[13](https://rdrr.io/cran/vegan/man/permutations.html) When the matrix dimension is small (\\( K < 7 \\)), it is customary to evaluate the criterion for all \\( K! \\) equally likely permutations; if \\( K \\) is large, a large number of random permutations are sampled with replacement.[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf) On how many permutations are needed there is a practical disagreement: worked examples and software defaults commonly use 999,[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/) while a methodological study of probability stability found that 500–2000 permutations can carry 5–6% error and recommended a minimum of 10,000 permutations, and 100,000 when the observed probability approaches a critical value such as 0.05.\n\n## Origin\n\nThe test was used to examine spatial patterns in the occurrence of leukemia.[14](https://aacrjournals.org/cancerres/article-pdf/27/2_Part_1/209/2382183/cr0272p10209.pdf) • [4](https://uw.pressbooks.pub/appliedmultivariatestatistics/chapter/mantel-test/) The application related a matrix of spatial distances and a matrix of temporal distances for the \\( n(n - 1)/2 \\) pairs formed from \\( n \\) observed disease cases, in a generalized regression approach.[14](https://aacrjournals.org/cancerres/article-pdf/27/2_Part_1/209/2382183/cr0272p10209.pdf) • [15](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425) He concluded the article by claiming the method was general for assessing correlation between entries of two square distance matrices.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)\n\nThe procedure was expanded to a nonparametric form of analysis.[15](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425) Assessing the statistic through a standard normal deviate, SND = Zm/var(Zm)^(1/2), is biased and works well only for large sample sizes; instead, the null distribution should be obtained empirically by permuting rows and columns of one distance matrix.[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/) The matrix correspondence test was subsequently adopted in population biology, after earlier use in geography and psychometrics.[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf) • [7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)\n\n## Variants\n\nSeveral derived forms operate on the same distance-matrix framework.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)\n\n- **Partial Mantel test.** Proposed by Smouse, Long, and Sokal (1986), it assesses the dependence between two distance matrices while controlling for a third, which may contain phylogenetic, geographic, or cost-distance values; the statistic is the partial correlation coefficient of the two distance variables given the third, with the same permutation procedure.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018) Smouse, Long, and Sokal showed that the Mantel treatment is really a regression analysis, with the normalized measure equivalent to a normalization of the cross-product statistic.[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)\n- **Mantel correlogram.** Borcard and Legendre (2012) warn against the partial Mantel test and recommend the Mantel correlogram instead.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) • [3](https://vegandevs.github.io/vegan/reference/mantel.html)\n- **Multiple regression on distance matrices (MRM).** With significance tests described in Legendre et al. (1994), MRM extends the classical and partial Mantel tests by regressing a response distance matrix on any number of explanatory distance matrices; it has attracted attention because it handles multiple explanatory variables, nonparametric relationships, many data types, and spatial autocorrelation at different scales.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) • [1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)\n- **CADM.** The test of congruence among distance matrices extends the framework to several matrices.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)\n- **Spearman variant.** The Spearman rather than Pearson correlation between the matrix entries can be computed.[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf) • [2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)\n\n## Applications\n\nSince its original use for the spatiotemporal dispersion of diseases, the Mantel test has been applied across the life sciences.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/) In spatial genetic analysis its simplest form compares geographic distances with genetic dissimilarities among the same objects.[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) In phylogenetic comparative contexts, the distance matrices are typically patristic distances, the sum of branch lengths separating pairs of species across a molecular phylogeny.[12](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)\n\n## Limitations and alternatives\n\nThe central limitation is dependence among the pairwise distances, which is why the permutation procedure exists; the [Monte Carlo](https://www.edgepedia.org/monte-carlo) null distribution handles this lack of independence for the simple test.[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf) The more serious problem is spatial autocorrelation. Guillot and Rousset showed that for the highest spatial scale parameter they considered (\\( \\kappa = 0.7 \\)), the Type I error rate actually achieved by a test targeting \\( \\alpha = 5\\% \\) ranges between 25% and 40%, increasing with sample size up to 55% when \\( n = 200 \\).[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018) They also found that the partial Mantel test fails to control autocorrelation: including a matrix of geographic distances does not remove the excess of small p-values, and the failure is not due to the choice of permutation variant.[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018) A controversy about the validity of the permutation procedure used in partial Mantel tests arose in the literature and is summarized in Legendre and Fortin (2010).[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf) Some Mantel-based approaches have shown inflated Type I error rates, for which Oden and Sokal (1992) pointed out a simple solution.[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/) Legendre, Fortin and Borcard (2015) question whether the Mantel test should be used in spatial analysis at all.[15](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425) A later benchmarking study confirmed that when spatial autocorrelation is present in both the response and at least one explanatory variable, the Type I error rate of the Mantel test, the partial Mantel test, and MRM increases with the intensity of autocorrelation and the number of autocorrelated explanatory variables.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/) It also found that the Mantel test and the partial Mantel test are not affected by inflated Type I error when there is no spatial autocorrelation, or when autocorrelation affects a single variable.[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)\n\nAgainst alternatives, a simulation comparison of distance-based methods found that ANOSIM and the Mantel test were very sensitive to heterogeneity in multivariate dispersions, with ANOSIM generally more sensitive, while [PERMANOVA](https://www.edgepedia.org/permanova) and Pillai's trace were largely unaffected for [balanced designs](https://www.edgepedia.org/balanced-design); for unbalanced designs all tests were too liberal when the smaller group had greater dispersion and overly conservative when the larger group had greater dispersion.[16](https://esajournals.onlinelibrary.wiley.com/doi/10.1890/12-2010.1) In simulations based on real ecological data sets, PERMANOVA was generally more powerful to detect changes in community structure, and the Mantel test was usually more powerful than ANOSIM.[16](https://esajournals.onlinelibrary.wiley.com/doi/10.1890/12-2010.1) Harmon and Glor (2010) documented poor statistical performance of the Mantel test specifically in phylogenetic comparative analyses using patristic distances.[12](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf) A generalised distance covariance (dCov) test has been proposed as a more flexible option that includes the RV coefficient and dCov as special cases and links to PROTEST.[17](https://onlinelibrary.wiley.com/doi/10.1002/env.2238)\n\n## References\n\n1. [Benchmarking the Mantel test and derived methods for testing association between distance matrices](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)\n2. [Comparison of the Mantel test and alternative approaches for detecting complex multivariate relationships in the spatial analysis of genetic data (Legendre & Fortin 2010, Molecular Ecology)](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)\n3. [Mantel and Partial Mantel Tests for Dissimilarity Matrices, vegan documentation](https://vegandevs.github.io/vegan/reference/mantel.html)\n4. [Mantel Test – Applied Multivariate Statistics in R](https://uw.pressbooks.pub/appliedmultivariatestatistics/chapter/mantel-test/)\n5. [Comparison of permutation methods for the partial correlation and partial Mantel tests (Legendre, Lapointe & Casgrain)](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)\n6. [Multiple Regression and Correlation Extensions of the Mantel Test of Matrix Correspondence (Smouse, Long & Sokal, 1986)](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)\n7. [Mantel test in population genetics (Diniz-Filho et al., 2013)](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)\n8. [Are probability estimates from the permutation model of Mantel's test stable?](https://doi.org/10.1139/z89-108)\n9. [Dismantling the Mantel tests (Guillot & Rousset, Methods in Ecology and Evolution)](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)\n10. [mantel (ecodist), CRAN reference manual](https://search.r-project.org/CRAN/refmans/ecodist/html/mantel.html)\n11. [Multiple regression on distance matrices: a multivariate spatial analysis tool (Lichstein 2007, Plant Ecology)](https://people.clas.ufl.edu/jlichstein/files/Lichstein_2007_Plant_Ecol.pdf)\n12. [Poor statistical performance of the Mantel test in phylogenetic comparative analyses (Harmon & Glor, 2010, Evolution)](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)\n13. [permutations: Permutation tests in Vegan](https://rdrr.io/cran/vegan/man/permutations.html)\n14. [The Detection of Disease Clustering and a Generalized Regression Approach (Cancer Research 27:209–220, 1967)](https://aacrjournals.org/cancerres/article-pdf/27/2_Part_1/209/2382183/cr0272p10209.pdf)\n15. [Should the Mantel test be used in spatial analysis? (Legendre, 2015, Methods in Ecology and Evolution)](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425)\n16. [Warton, Wright & Wang (2012), Distance-based multivariate analyses confounded by multivariate dispersion, simulation comparison of ANOSIM, PERMANOVA and the Mantel test](https://esajournals.onlinelibrary.wiley.com/doi/10.1890/12-2010.1)\n17. [A comparison of the Mantel test with a generalised distance covariance test (Environmetrics)](https://onlinelibrary.wiley.com/doi/10.1002/env.2238)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Multivariate association and dimension reduction › Titles A to Me*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/mantel-test", "markdown_url": "https://www.edgepedia.org/mantel-test.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Mantel test\", Edgepedia (EdgeChat), https://www.edgepedia.org/mantel-test. Edgepedia Community License 1.0.", "credit_md": "\"[Mantel test](https://www.edgepedia.org/mantel-test)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/mantel-test](https://www.edgepedia.org/mantel-test). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Mantel test\", Edgepedia (EdgeChat), https://www.edgepedia.org/mantel-test. Edgepedia Community License 1.0.", "speakable": "The Mantel test is a permutation-based statistical test that assesses the association between two distance matrices for the same objects, such as geographic and genetic distances." }