{ "id": "ep42sg06hp", "slug": "leave-one-out-cross-validation", "title": "Leave-one-out cross-validation", "updated": "2026-10-15", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics" }, { "id": "physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference" } ], "geo": [ { "id": "geo.nongeo.t1946.physical.math.statistics.stats_statistical_inference", "label": "Non-geographic · 1946 to 2000: Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.statistics.stats_statistical_inference", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical" }, { "id": "geo.nongeo.t1946.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math" }, { "id": "geo.nongeo.t1946.physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.statistics" }, { "id": "geo.nongeo.t1946.physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.math.statistics.stats_statistical_inference" } ] } ], "excerpt": "Leave-one-out cross-validation (LOOCV) is a resampling method that estimates a model's out-of-sample prediction error by repeatedly leaving out one observation, equivalent to k-fold cross-validation with k equal to the sample size.", "snippet": "Leave-one-out cross-validation (LOOCV) is a resampling method that estimates a model's out-of-sample prediction error by repeatedly leaving out one observation, equivalent to k-fold cross-validation with k equal to the sample size.", "node": "physical.math.statistics.stats_statistical_inference", "markdown": "# Leave-one-out cross-validation\n\nLeave-one-out cross-validation (LOOCV) is a resampling method that estimates a model's out-of-sample prediction error by repeatedly fitting the model on all observations except one and evaluating it on the omitted observation. It is the special case of k-fold cross-validation with k equal to the sample size n, and it is used to estimate generalization error and to select models or tuning parameters.[1](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html)\n\n| Key fact | Detail |\n|---|---|\n| What it produces | The average of n held-out losses, one per observation, each from a model fitted on the other \\( n - 1 \\) points[2](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False) |\n| Relation to k-fold | Equivalent to `KFold(n_splits=n)` and `LeavePOut(p=1)` in scikit-learn[1](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html) |\n| Bias | Pessimistic (overestimates error) with bias \\( o(n^{-1}) > 0 \\), smaller than plug-in and K-fold estimates[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/ac4106bcfff33140de7799d03daeb8a4-Paper-Conference.pdf) |\n| Cost | n model refits; very costly for large datasets unless a closed-form shortcut exists[1](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html) |\n| Efficient cases | Linear and ridge regression (hat matrix), regularized GLMs (ALO), Bayesian models (PSIS-LOO)[4](https://engineering.purdue.edu/MakinLab/IMSL/A1.S1.html) • [5](https://par.nsf.gov/servlets/purl/10183412) • [6](https://stat.columbia.edu/~gelman/research/published/loo_stan.pdf) |\n| Known failure modes | Distributional bias in the aggregated training sets, biased c-statistics[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC12662204/) • [8](https://pmc.ncbi.nlm.nih.gov/articles/PMC10152625/) |\n\n## How it works\n\nThe estimator is an average over n single-observation deletions: for a learning algorithm A and data set \\( D_n \\) of size n, the LOO estimate is\n\n\\[ \\mathrm{LOO}(A; D_n) = \\frac{1}{n} \\sum_{j=1}^{n} \\gamma\\bigl(A(D_n^{(-j)}); \\xi_j\\bigr), \\]\n\nwhere \\( D_n^{(-j)} \\) is the data with observation j removed and γ is the loss measured on the held-out point \\( \\xi_j \\).[2](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False) Each data point serves once as validation data, so the estimate tracks the risk of a model trained on \\( n - 1 \\) observations rather than n; the bias of any cross-validation estimator is exactly this difference in risks, which is nonnegative and shrinks as the training portion grows.[2](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False) A 2024 analysis gives the rate: LOOCV bias is \\( o(n^{-1}) \\) and positive, smaller than that of plug-in and K-fold estimators, whose bias is Θ(K⁻¹n⁻²γ) for suitable exponents γ.[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/ac4106bcfff33140de7799d03daeb8a4-Paper-Conference.pdf) Asymptotically, the variance of V-fold cross-validation decreases with V, so LOO has the minimal variance among V-fold estimators.[2](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False) The often-repeated claim that LOOCV has much higher variance than 5-fold or 10-fold CV is described as more or less heuristic; several comparative studies found similar variances, and no universal unbiased estimator of the variance of [K-fold cross-validation](https://www.edgepedia.org/k-fold-cross-validation) exists, even for \\( K = n \\).[9](https://par.nsf.gov/servlets/purl/10329391) • [10](https://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf)\n\n## How it is done\n\nA practitioner loops over the n observations. For each j, the model is refitted on the \\( n - 1 \\) remaining observations, a prediction is made for the held-out point, and the loss γ is recorded. The LOOCV estimate is the mean of the n losses. In scikit-learn this is `LeaveOneOut`, which generates n singleton test sets.[1](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html)\n\nTwo pitfalls matter in practice. First, any feature selection or preprocessing that uses the response must be redone inside each refit; performing it once on the full data before splitting introduces additional bias.[11](https://brb.nci.nih.gov/techreport/TechReport_Molinaro.pdf) Second, the n training sets produced by LOOCV are not identically distributed in a subtle way: under LOOCV a perfect negative correlation (\\( r = -1 \\)) emerges between the mean label of each training set and the label of its held-out test instance, a phenomenon called distributional bias that also affects regression.[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC12662204/)\n\n## Origin\n\nThe modern formulation took shape in the mid-1970s. Stone's 1974 paper in the Journal of the Royal Statistical Society Series B applied a generalized form of the cross-validation criterion to the choice and assessment of statistical predictions.[12](https://rss.onlinelibrary.wiley.com/doi/10.1111/j.2517-6161.1974.tb00994.x) Geisser published \"A predictive approach to the random effect model\" in [Biometrika](https://www.edgepedia.org/biometrika) in 1974,[13](https://doi.org/10.1093/biomet/61.1.101) and PRESS (predicted residual sum of squares), introduced by David M. Allen in 1974, appeared that year as well. Survey literature on cross-validation treats these mid-1970s papers as the point where leave-one-out validation was independently formulated, and notes that simple hold-out validation predates them by decades.[2](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False) The generalized cross-validation (GCV) criterion, a rotation-invariant relative of PRESS, appears in Golub, Heath, and Wahba's 1979 Technometrics paper on choosing a ridge parameter.[14](https://doi.org/10.1080/00401706.1979.10489751)\n\n## Variants\n\nNaive LOOCV costs n refits, but several model classes admit closed-form shortcuts.\n\n**Linear models.** For least-squares regression the hat matrix diagonal \\( h_{nn} \\) measures how much weight observation \\( y_n \\) gets in predicting itself, and a Sherman–Morrison-based update gives the LOO prediction in one step, with the factor \\( 1 - h_{nn} \\) in the denominator.[4](https://engineering.purdue.edu/MakinLab/IMSL/A1.S1.html) For ridge regression, Golub, Heath, and Wahba's formula computes the exact LOOCV error at essentially the cost of a single ridge fit,[9](https://par.nsf.gov/servlets/purl/10329391) and a Sherman–Morrison–Woodbury shortcut carries out LOOCV exactly in the time needed to fit a small number of base ridge regressions.[15](https://arxiv.org/pdf/2007.12671)\n\n**Regularized estimators.** Approximate leave-one-out (ALO), introduced by Rahnama Rad and Maleki in 2020 in the Journal of the Royal Statistical Society Series B, computes a closed-form LOO surrogate for lasso, elastic net, bridge, and ridge-type logistic or [Poisson regression](https://www.edgepedia.org/poisson-regression) from a single optimization solve, one matrix inversion, and two matrix–matrix multiplications; the bottleneck is inverting the generalized hat matrix H.[5](https://par.nsf.gov/servlets/purl/10183412) In the high-dimensional regime where \\( n/p \\) tends to a constant, \\( |\\mathrm{ALO} - \\mathrm{LOO}| = O_p(\\mathrm{poly}\\log n / \\sqrt{n}) \\).[16](https://proceedings.mlr.press/v108/rad20a/rad20a.pdf) ALO has been shown consistent for ℓ₁ regularizers even in high-dimensional settings without sparsity assumptions,[17](https://doi.org/10.1109/tit.2024.3450002) and RandALO, a randomized version of ALO, is a consistent high-dimensional risk estimator that costs less than K-fold CV, using randomized numerical linear algebra to reduce the work to a constant number of quadratic programs; ALO itself generalizes GCV and coincides exactly with LOO for ridge regression.[18](https://arxiv.org/html/2409.09781v1)\n\n**Bayesian models.** PSIS-LOO, introduced by Vehtari, Gelman, and Gabry in 2016 in [Statistics and Computing](https://www.edgepedia.org/statistics-and-computing), reuses existing posterior draws and applies Pareto-smoothed importance sampling, fitting a generalized [Pareto distribution](https://www.edgepedia.org/pareto-distribution) to the 20% largest importance ratios for each held-out point, to estimate elpd_loo = Σᵢ log p(yᵢ|y₋ᵢ).[6](https://stat.columbia.edu/~gelman/research/published/loo_stan.pdf) For Bayesian non-factorized multivariate normal and Student-t models, Bürkner, Gabry, and Vehtari showed in 2020 in Computational Statistics that exact and approximate LOO-CV can be computed efficiently; models with the marginalization property (such as Gaussian processes) simply drop a row and column of the covariance, while spatial-lag models treat the held-out value as missing.[19](https://link.springer.com/article/10.1007/s00180-020-01045-4)\n\n## Applications\n\nLOOCV is used to estimate prediction error and to choose models or tuning parameters. GCV, its rotation-invariant relative, minimizes V(λ) = (1/n)‖y − A(λ)y‖² / [(1/n)tr(I − A(λ))]² with A(λ) = X(X'X + nλI)⁻¹X' to select the ridge parameter, and remains usable when \\( n - p \\) is small or even when \\( p > n \\).[20](https://pages.stat.wisc.edu/~wahba/stat860public/pdf1/golub.heath.wahba.pdf) In small high-dimensional genomic studies, LOOCV, 10-fold CV, and the .632+ bootstrap give the smallest bias among resampling schemes for diagonal discriminant analysis, nearest neighbors, and classification trees.[11](https://brb.nci.nih.gov/techreport/TechReport_Molinaro.pdf) In Bayesian workflow, \\( \\mathrm{elpd}_{\\mathrm{loo}} \\) from PSIS-LOO supports model comparison; LOO and WAIC are both [consistent estimators](https://www.edgepedia.org/consistent-estimator) of the expected log predictive density, but LOO has been found more robust with outliers or weak priors.[6](https://stat.columbia.edu/~gelman/research/published/loo_stan.pdf)\n\n## Limitations and alternatives\n\nThe main cost is n refits, which for large datasets makes K-fold CV, with K typically 5 to 10, the popular substitute.[1](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html) • [3](https://proceedings.neurips.cc/paper_files/paper/2024/file/ac4106bcfff33140de7799d03daeb8a4-Paper-Conference.pdf) K-fold CV estimates the risk of a model trained on \\( n(K-1)/K \\) points, a bias that vanishes only as K approaches n.[18](https://arxiv.org/html/2409.09781v1) Performance-measure choice matters: LOO cross-validated c-statistics can be strongly biased toward zero because the estimated probability for a left-out event is on average lower than for a left-out non-event, and the bias is more severe for methods that shrink probabilities, such as ridge regression; the [Brier score](https://www.edgepedia.org/brier-score) is nearly unbiased under LOO while the discrimination slope is pessimistically biased. Leave-pair-out CV performs better for the c-statistic but needs \\( (n-k) \\cdot k \\) fits instead of n.[8](https://pmc.ncbi.nlm.nih.gov/articles/PMC10152625/) Bootstrap alternatives include the .632 and .632+ estimators, which combine the resubstitution error with the leave-one-out bootstrap (out-of-bag) error, with .632+ adjusting the weight to correct for overfitting.[2](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False) The naive plug-in LOOCV estimate of the error of a CV-tuned model is biased, and the honest LOOCV (HLOOCV) estimator of Wang and Zou, published in Stat in 2021, corrects this for post-tuning generalization error.[9](https://par.nsf.gov/servlets/purl/10329391) The distributional-bias analysis shows that aggregating LOOCV predictions biases hyperparameter optimization toward weaker regularization, and proposes RLOOCV, which removes a randomly selected opposite-label instance from each training set to restore matching train and test distributions; post hoc normalization of predictions is not a valid fix, since it overestimates performance for small-K nearest-neighbor models.[7](https://pmc.ncbi.nlm.nih.gov/articles/PMC12662204/) A NeurIPS 2024 regime analysis finds that for nonparametric models with slow rates (\\( \\gamma \\cdot v \\le 1/4 \\), including k-nearest neighbors with k_n = Θ(√n)), only LOOCV has negligible bias and provides valid interval coverage, while for fast rates plug-in estimation is computationally preferable.[3](https://proceedings.neurips.cc/paper_files/paper/2024/file/ac4106bcfff33140de7799d03daeb8a4-Paper-Conference.pdf)\n\n## References\n\n1. [LeaveOneOut, scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html)\n2. [A survey of cross-validation procedures for model selection (Arlot & Celisse, 2010)](https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2F09-SS054&isResultClick=False)\n3. [Is Cross-Validation the Gold Standard to Estimate Out-of-sample Model Performance? (NeurIPS 2024)](https://proceedings.neurips.cc/paper_files/paper/2024/file/ac4106bcfff33140de7799d03daeb8a4-Paper-Conference.pdf)\n4. [Leave-one-out cross validation for linear regression in one step (An Introduction to Modern Statistical Learning, Appendix A)](https://engineering.purdue.edu/MakinLab/IMSL/A1.S1.html)\n5. [A scalable estimate of the out-of-sample prediction error via approximate leave-one-out cross-validation (Rahnama Rad & Maleki, JRSS-B)](https://par.nsf.gov/servlets/purl/10183412)\n6. [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC (Vehtari, Gelman, Gabry)](https://stat.columbia.edu/~gelman/research/published/loo_stan.pdf)\n7. [Distributional bias compromises leave-one-out cross-validation (Science Advances; PMC copy)](https://pmc.ncbi.nlm.nih.gov/articles/PMC12662204/)\n8. [Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures, a simulation study](https://pmc.ncbi.nlm.nih.gov/articles/PMC10152625/)\n9. [Honest leave-one-out cross-validation for estimating post-tuning generalization error (Wang and Zou line of work)](https://par.nsf.gov/servlets/purl/10329391)\n10. [No Unbiased Estimator of the Variance of K-Fold Cross-Validation (Bengio and Grandvalet, JMLR 2004)](https://www.jmlr.org/papers/volume5/grandvalet04a/grandvalet04a.pdf)\n11. [Prediction error estimation: a comparison of resampling methods for high-dimensional genomic data (Molinaro et al., NCI technical report)](https://brb.nci.nih.gov/techreport/TechReport_Molinaro.pdf)\n12. [Cross-Validatory Choice and Assessment of Statistical Predictions (Stone, 1974)](https://rss.onlinelibrary.wiley.com/doi/10.1111/j.2517-6161.1974.tb00994.x)\n13. [SEYMOUR GEISSER (1974). A predictive approach to the random effect model. Biometrika.](https://doi.org/10.1093/biomet/61.1.101)\n14. [Gene H. Golub, Michael Heath, Grace Wahba (1979). Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics.](https://doi.org/10.1080/00401706.1979.10489751)\n15. [Cross-validation confidence intervals for model selection (CLT paper)](https://arxiv.org/pdf/2007.12671)\n16. [A theoretical analysis of leave-one-out cross-validation in high-dimensional settings (PMLR v108)](https://proceedings.mlr.press/v108/rad20a/rad20a.pdf)\n17. [Arnab Auddy and colleagues (2024). Approximate Leave-One-Out Cross Validation for Regression With ℓ₁ Regularizers. IEEE Transactions on Information Theory.](https://doi.org/10.1109/tit.2024.3450002)\n18. [RandALO: Out-of-sample risk estimation in no time flat (2024)](https://arxiv.org/html/2409.09781v1)\n19. [Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models (Bürkner, Gabry, Vehtari)](https://link.springer.com/article/10.1007/s00180-020-01045-4)\n20. [Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter (Golub, Heath & Wahba, Technometrics, 1979)](https://pages.stat.wisc.edu/~wahba/stat860public/pdf1/golub.heath.wahba.pdf)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Statistical inference, estimation, sampling, and testing*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://stat.columbia.edu/~gelman/research/published/loo_stan.pdf", "https://pages.stat.wisc.edu/~wahba/stat860public/pdf1/golub.heath.wahba.pdf" ], "url": "https://www.edgepedia.org/leave-one-out-cross-validation", "markdown_url": "https://www.edgepedia.org/leave-one-out-cross-validation.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Leave-one-out cross-validation\", Edgepedia (EdgeChat), https://www.edgepedia.org/leave-one-out-cross-validation. Edgepedia Community License 1.0.", "credit_md": "\"[Leave-one-out cross-validation](https://www.edgepedia.org/leave-one-out-cross-validation)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/leave-one-out-cross-validation](https://www.edgepedia.org/leave-one-out-cross-validation). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Leave-one-out cross-validation\", Edgepedia (EdgeChat), https://www.edgepedia.org/leave-one-out-cross-validation. Edgepedia Community License 1.0.", "speakable": "Leave-one-out cross-validation is a resampling method that estimates a model's out-of-sample prediction error by repeatedly leaving out one observation, equivalent to k-fold cross-validation with k equal to the sample size." }