# Mantel test

The Mantel test is a permutation-based statistical test that assesses the association between two square matrices whose entries are distances, dissimilarities, or similarities among the same n objects. It has been applied in different fields of the life sciences.<sup>[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)</sup> In its simplest form it compares a matrix of geographic distances to a matrix of genetic or other dissimilarities among the same objects.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup> Because the pairwise entries of a distance matrix are mutually dependent, the usual significance tests are invalid, and the test instead builds its null distribution by permuting whole objects.<sup>[3](https://vegandevs.github.io/vegan/reference/mantel.html)</sup>

| Key fact | Detail |
|---|---|
| Statistic | The Mantel correlation is the Pearson correlation between the corresponding upper-triangular entries of two distance matrices; it ranges from −1 to +1.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup><sup> • </sup><sup>[4](https://uw.pressbooks.pub/appliedmultivariatestatistics/chapter/mantel-test/)</sup> |
| Null hypothesis | Distances among objects in matrix A are linearly independent of distances among the same objects in matrix B.<sup>[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)</sup> |
| Randomization | One matrix is held rigid while the rows and corresponding columns of the other are permuted together, preserving matrix symmetry.<sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup> |
| Permutation count | 999 permutations are common in worked examples, but probability estimates from 500–2000 permutations can carry 5–6% error; 10,000 or more are recommended for stable p-values.<sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup><sup> • </sup><sup>[8](https://doi.org/10.1139/z89-108)</sup> |
| Main variants | Partial Mantel test, Mantel correlogram, multiple regression on distance matrices (MRM), and CADM.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup> |
| Known failure mode | Under spatial autocorrelation in both response and explanatory variables, Type I error rates can reach 25–55% at a nominal 5% level.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup> |
| Software | The vegan package computes the Mantel and partial Mantel tests; ecodist provides the partial Mantel r and MRM.<sup>[3](https://vegandevs.github.io/vegan/reference/mantel.html)</sup><sup> • </sup><sup>[10](https://search.r-project.org/CRAN/refmans/ecodist/html/mantel.html)</sup> |

## How it works

The Mantel test is a test for association between two \( n \times n \) matrices whose entries are distances or similarities between all pairs of objects.<sup>[11](https://people.clas.ufl.edu/jlichstein/files/Lichstein_2007_Plant_Ecol.pdf)</sup> His original statistic was the cross-product of the distances in the two matrices, that is, the sum over all pairs of the product of the two distance values.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup><sup> • </sup><sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup> Modern programs instead compute the Mantel correlation, the cross-product between standardized distances divided by \( (d - 1) \), where \( d \) is the number of distances in the upper-triangular portion of each matrix; this is simply the Pearson correlation between the corresponding entries, bounded between −1 and +1, and the transformation has no effect on the permutation-test probability.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup><sup> • </sup><sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup>

The null hypothesis is that the distances among objects in matrix A are linearly independent of the distances among the same objects in matrix B.<sup>[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)</sup> For symmetric matrices only the upper (or lower) triangular entries are used; for non-symmetric matrices both portions are included, and the main diagonal need not be included.<sup>[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)</sup>

The reason permutation is required is structural: an \( n \times n \) distance matrix contains \( n(n - 1)/2 \) entries for only \( n \) observations, so the entries are mutually dependent and the usual significance tests are invalid.<sup>[3](https://vegandevs.github.io/vegan/reference/mantel.html)</sup><sup> • </sup><sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup> Under the null hypothesis, the objects, not the distances, are the permutable units.<sup>[5](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)</sup> Randomization therefore permutes the n objects of one matrix, which is equivalent to permuting its rows and corresponding columns together: if rows one and four are swapped, columns one and four are also exchanged, so the matrix stays symmetric.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup><sup> • </sup><sup>[12](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)</sup>

## How it is done

The procedure has four steps.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup>

1. Build the two \( n \times n \) distance matrices from the same set of objects.
2. Compute the scalar product (or equivalently the Mantel correlation) between the corresponding entries.
3. For a large number of iterations, draw a [random permutation](https://www.edgepedia.org/random-permutation) of {1, …, n} uniformly, apply it to the rows and corresponding columns of one matrix, and recompute the statistic; one matrix is held rigid while the other is permuted.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup><sup> • </sup><sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup>
4. Derive the p-value by comparing the observed statistic with the permuted values, typically from the upper tail for a one-sided test of positive correlation.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup><sup> • </sup><sup>[3](https://vegandevs.github.io/vegan/reference/mantel.html)</sup>

With 999 randomizations, if none of the randomized values exceeds the observed one, the p-value is reported as \( 1/1000 \), since the observed value is conservatively added to both numerator and denominator.<sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup> More generally the permutation p-value is \( p = (N + 1)/(n + 1) \), where \( n \) is the number of permutations, so the minimum achievable p-value is \( 1/(n + 1) \).<sup>[13](https://rdrr.io/cran/vegan/man/permutations.html)</sup> When the matrix dimension is small (\( K < 7 \)), it is customary to evaluate the criterion for all \( K! \) equally likely permutations; if \( K \) is large, a large number of random permutations are sampled with replacement.<sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup> On how many permutations are needed there is a practical disagreement: worked examples and software defaults commonly use 999,<sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup> while a methodological study of probability stability found that 500–2000 permutations can carry 5–6% error and recommended a minimum of 10,000 permutations, and 100,000 when the observed probability approaches a critical value such as 0.05.

## Origin

The test was used to examine spatial patterns in the occurrence of leukemia.<sup>[14](https://aacrjournals.org/cancerres/article-pdf/27/2_Part_1/209/2382183/cr0272p10209.pdf)</sup><sup> • </sup><sup>[4](https://uw.pressbooks.pub/appliedmultivariatestatistics/chapter/mantel-test/)</sup> The application related a matrix of spatial distances and a matrix of temporal distances for the \( n(n - 1)/2 \) pairs formed from \( n \) observed disease cases, in a generalized regression approach.<sup>[14](https://aacrjournals.org/cancerres/article-pdf/27/2_Part_1/209/2382183/cr0272p10209.pdf)</sup><sup> • </sup><sup>[15](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425)</sup> He concluded the article by claiming the method was general for assessing correlation between entries of two square distance matrices.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup>

The procedure was expanded to a nonparametric form of analysis.<sup>[15](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425)</sup> Assessing the statistic through a standard normal deviate, SND = Zm/var(Zm)^(1/2), is biased and works well only for large sample sizes; instead, the null distribution should be obtained empirically by permuting rows and columns of one distance matrix.<sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup> The matrix correspondence test was subsequently adopted in population biology, after earlier use in geography and psychometrics.<sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup><sup> • </sup><sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup>

## Variants

Several derived forms operate on the same distance-matrix framework.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup>

- **Partial Mantel test.** Proposed by Smouse, Long, and Sokal (1986), it assesses the dependence between two distance matrices while controlling for a third, which may contain phylogenetic, geographic, or cost-distance values; the statistic is the partial correlation coefficient of the two distance variables given the third, with the same permutation procedure.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup> Smouse, Long, and Sokal showed that the Mantel treatment is really a regression analysis, with the normalized measure equivalent to a normalization of the cross-product statistic.<sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup>
- **Mantel correlogram.** Borcard and Legendre (2012) warn against the partial Mantel test and recommend the Mantel correlogram instead.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup><sup> • </sup><sup>[3](https://vegandevs.github.io/vegan/reference/mantel.html)</sup>
- **Multiple regression on distance matrices (MRM).** With significance tests described in Legendre et al. (1994), MRM extends the classical and partial Mantel tests by regressing a response distance matrix on any number of explanatory distance matrices; it has attracted attention because it handles multiple explanatory variables, nonparametric relationships, many data types, and spatial autocorrelation at different scales.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup><sup> • </sup><sup>[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)</sup>
- **CADM.** The test of congruence among distance matrices extends the framework to several matrices.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup>
- **Spearman variant.** The Spearman rather than Pearson correlation between the matrix entries can be computed.<sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup><sup> • </sup><sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup>

## Applications

Since its original use for the spatiotemporal dispersion of diseases, the Mantel test has been applied across the life sciences.<sup>[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)</sup> In spatial genetic analysis its simplest form compares geographic distances with genetic dissimilarities among the same objects.<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup> In phylogenetic comparative contexts, the distance matrices are typically patristic distances, the sum of branch lengths separating pairs of species across a molecular phylogeny.<sup>[12](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)</sup>

## Limitations and alternatives

The central limitation is dependence among the pairwise distances, which is why the permutation procedure exists; the [Monte Carlo](https://www.edgepedia.org/monte-carlo) null distribution handles this lack of independence for the simple test.<sup>[6](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)</sup> The more serious problem is spatial autocorrelation. Guillot and Rousset showed that for the highest spatial scale parameter they considered (\( \kappa = 0.7 \)), the Type I error rate actually achieved by a test targeting \( \alpha = 5\% \) ranges between 25% and 40%, increasing with sample size up to 55% when \( n = 200 \).<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup> They also found that the partial Mantel test fails to control autocorrelation: including a matrix of geographic distances does not remove the excess of small p-values, and the failure is not due to the choice of permutation variant.<sup>[9](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)</sup> A controversy about the validity of the permutation procedure used in partial Mantel tests arose in the literature and is summarized in Legendre and Fortin (2010).<sup>[2](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)</sup> Some Mantel-based approaches have shown inflated Type I error rates, for which Oden and Sokal (1992) pointed out a simple solution.<sup>[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)</sup> Legendre, Fortin and Borcard (2015) question whether the Mantel test should be used in spatial analysis at all.<sup>[15](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425)</sup> A later benchmarking study confirmed that when spatial autocorrelation is present in both the response and at least one explanatory variable, the Type I error rate of the Mantel test, the partial Mantel test, and MRM increases with the intensity of autocorrelation and the number of autocorrelated explanatory variables.<sup>[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)</sup> It also found that the Mantel test and the partial Mantel test are not affected by inflated Type I error when there is no spatial autocorrelation, or when autocorrelation affects a single variable.<sup>[1](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)</sup>

Against alternatives, a simulation comparison of distance-based methods found that ANOSIM and the Mantel test were very sensitive to heterogeneity in multivariate dispersions, with ANOSIM generally more sensitive, while [PERMANOVA](https://www.edgepedia.org/permanova) and Pillai's trace were largely unaffected for [balanced designs](https://www.edgepedia.org/balanced-design); for unbalanced designs all tests were too liberal when the smaller group had greater dispersion and overly conservative when the larger group had greater dispersion.<sup>[16](https://esajournals.onlinelibrary.wiley.com/doi/10.1890/12-2010.1)</sup> In simulations based on real ecological data sets, PERMANOVA was generally more powerful to detect changes in community structure, and the Mantel test was usually more powerful than ANOSIM.<sup>[16](https://esajournals.onlinelibrary.wiley.com/doi/10.1890/12-2010.1)</sup> Harmon and Glor (2010) documented poor statistical performance of the Mantel test specifically in phylogenetic comparative analyses using patristic distances.<sup>[12](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)</sup> A generalised distance covariance (dCov) test has been proposed as a more flexible option that includes the RV coefficient and dCov as special cases and links to PROTEST.<sup>[17](https://onlinelibrary.wiley.com/doi/10.1002/env.2238)</sup>

## References

1. [Benchmarking the Mantel test and derived methods for testing association between distance matrices](https://pmc.ncbi.nlm.nih.gov/articles/PMC11696488/)
2. [Comparison of the Mantel test and alternative approaches for detecting complex multivariate relationships in the spatial analysis of genetic data (Legendre & Fortin 2010, Molecular Ecology)](http://www.numericalecology.com/Reprints/Legendre_Fortin_MER_2010.pdf)
3. [Mantel and Partial Mantel Tests for Dissimilarity Matrices, vegan documentation](https://vegandevs.github.io/vegan/reference/mantel.html)
4. [Mantel Test – Applied Multivariate Statistics in R](https://uw.pressbooks.pub/appliedmultivariatestatistics/chapter/mantel-test/)
5. [Comparison of permutation methods for the partial correlation and partial Mantel tests (Legendre, Lapointe & Casgrain)](http://www.numericalecology.com/Reprints/Partial_Mantel_paper.pdf)
6. [Multiple Regression and Correlation Extensions of the Mantel Test of Matrix Correspondence (Smouse, Long & Sokal, 1986)](http://www.pelagicos.net/MARS6300/readings/Smouse_et_al._1986.pdf)
7. [Mantel test in population genetics (Diniz-Filho et al., 2013)](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873175/)
8. [Are probability estimates from the permutation model of Mantel's test stable?](https://doi.org/10.1139/z89-108)
9. [Dismantling the Mantel tests (Guillot & Rousset, Methods in Ecology and Evolution)](https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.12018)
10. [mantel (ecodist), CRAN reference manual](https://search.r-project.org/CRAN/refmans/ecodist/html/mantel.html)
11. [Multiple regression on distance matrices: a multivariate spatial analysis tool (Lichstein 2007, Plant Ecology)](https://people.clas.ufl.edu/jlichstein/files/Lichstein_2007_Plant_Ecol.pdf)
12. [Poor statistical performance of the Mantel test in phylogenetic comparative analyses (Harmon & Glor, 2010, Evolution)](https://lukejharmon.github.io/assets/HarmonGlor_2010_Evolution.pdf)
13. [permutations: Permutation tests in Vegan](https://rdrr.io/cran/vegan/man/permutations.html)
14. [The Detection of Disease Clustering and a Generalized Regression Approach (Cancer Research 27:209–220, 1967)](https://aacrjournals.org/cancerres/article-pdf/27/2_Part_1/209/2382183/cr0272p10209.pdf)
15. [Should the Mantel test be used in spatial analysis? (Legendre, 2015, Methods in Ecology and Evolution)](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12425)
16. [Warton, Wright & Wang (2012), Distance-based multivariate analyses confounded by multivariate dispersion, simulation comparison of ANOSIM, PERMANOVA and the Mantel test](https://esajournals.onlinelibrary.wiley.com/doi/10.1890/12-2010.1)
17. [A comparison of the Mantel test with a generalised distance covariance test (Environmetrics)](https://onlinelibrary.wiley.com/doi/10.1002/env.2238)

---
*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Multivariate association and dimension reduction › Titles A to Me*

*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license