{ "id": "epbgxrbb1r", "slug": "kernel-ridge-regression", "title": "Kernel ridge regression", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" }, { "id": "technology.computing.ai.supervised_unsupervised", "label": "Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.supervised_unsupervised" }, { "id": "technology.computing.ai.supervised_unsupervised.kernel_methods_svms", "label": "Kernel methods and support vector machines", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.supervised_unsupervised.kernel_methods_svms" } ], "geo": [ { "id": "geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised", "label": "Non-geographic · 1946 to 2000: Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology" }, { "id": "geo.nongeo.t1946.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing" }, { "id": "geo.nongeo.t1946.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_methods" }, { "id": "geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised", "label": "Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.supervised_unsupervised" } ] } ], "excerpt": "Kernel ridge regression (KRR) is a nonlinear regression method that combines ridge regression with the kernel trick, yielding a closed-form predictor used in molecular property prediction and genomic prediction.", "snippet": "Kernel ridge regression (KRR) is a nonlinear regression method that combines ridge regression with the kernel trick, yielding a closed-form predictor used in molecular property prediction and genomic prediction.", "node": "technology.computing.ai.supervised_unsupervised.kernel_methods_svms", "markdown": "# Kernel ridge regression\n\nKernel ridge regression (KRR) is a nonlinear regression method that solves L2-regularized least squares in a kernel-induced feature space, producing a closed-form predictor for a continuous target variable.[1](https://scikit-learn.org/stable/modules/kernel_ridge.html) It combines ridge regression, the linear least-squares estimator with an L2 penalty, with the kernel trick, so that for a nonlinear kernel the learned function is linear in the feature space but nonlinear in the original inputs.[1](https://scikit-learn.org/stable/modules/kernel_ridge.html) KRR is used for small-to-medium supervised regression problems, including molecular property prediction, healthcare data modeling, and genomic prediction.[2](https://www.jmlr.org/papers/v27/25-0385.html)\n\n| Key fact | Detail |\n|---|---|\n| Definition | Ridge regression (least squares with L2 regularization) combined with the kernel trick[1](https://scikit-learn.org/stable/modules/kernel_ridge.html) |\n| Closed-form solution | \\( \\alpha^{*} = (K + \\lambda I)^{-1} y \\), where \\( K \\) is the n × n kernel matrix[3](https://web2.qatar.cmu.edu/~gdicaro/10315-Fall19/additional/welling-notes-on-kernel-ridge.pdf) |\n| Exact cost | \\( O(n^{3}) \\) time and \\( O(n^{2}) \\) memory for n training samples[4](https://ar5iv.labs.arxiv.org/html/2509.02649) |\n| Sparsity | The kernel expansion is in general fully dense, unlike support vector regression[5](http://theoval.cmp.uea.ac.uk/publications/pdf/npl2002a.pdf) |\n| Relation to GPR | The KRR estimate coincides with the posterior mean of kriging (Gaussian process regression); GPR additionally outputs uncertainty[6](https://arxiv.org/html/2311.01762v2) • [7](https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_compare_gpr_krr.html) |\n| Typical kernels | RBF, Laplacian, Matérn-5/2, Gaussian[2](https://www.jmlr.org/papers/v27/25-0385.html) |\n| Approximations | Nyström and random features reduce cost to \\( O(n^{3/2}) \\) at the price of additional error[4](https://ar5iv.labs.arxiv.org/html/2509.02649) |\n\n## How it works\n\nKRR is formulated as regularized empirical risk minimization in a reproducing kernel [Hilbert space](https://www.edgepedia.org/hilbert-space) (RKHS), a function space associated with the chosen kernel. The estimator minimizes the sum of squared errors plus \\( \\lambda \\cdot \\|f\\|_{\\mathcal{H}}^{2} \\), where \\( \\lambda > 0 \\) tunes the trade-off between fit to the data and model complexity.[8](https://mlweb.loria.fr/book/en/kernelridgeregression.html)\n\nBy the representer theorem, the solution has the form \\( f = \\sum_{i=1}^{n} \\alpha_{i} \\cdot K(\\mathbf{x}_{i}, \\cdot) \\), which reduces the infinite-dimensional problem to a finite one in the coefficients \\( \\alpha \\).[8](https://mlweb.loria.fr/book/en/kernelridgeregression.html) The dual objective is \\( \\|\\mathbf{y} - \\mathbf{K} \\cdot \\boldsymbol{\\alpha}\\|_2^2 + \\lambda \\cdot \\boldsymbol{\\alpha}^{T} \\cdot \\mathbf{K} \\cdot \\boldsymbol{\\alpha} \\), and setting its gradient to zero gives the closed form \\( \\alpha^{*} = (K + \\lambda I)^{-1} y \\).[8](https://mlweb.loria.fr/book/en/kernelridgeregression.html) • [3](https://web2.qatar.cmu.edu/~gdicaro/10315-Fall19/additional/welling-notes-on-kernel-ridge.pdf) Prediction is then \\( \\hat{f}(x) = \\sum_{i=1}^{n} \\alpha_{i} \\cdot k(x, x_{i}) \\), unique for \\( \\lambda > 0 \\).[9](https://ar5iv.labs.arxiv.org/html/1807.02582)\n\nThe kernel trick is what makes this nonlinear: kernel functions represent dot products in a feature space, so the algorithm operates in that space without computing coordinates in it.[10](https://media.wix.com/ugd/5256f1_27f0134dd65d4b528c5833b054ba2669.pdf) Regularization is necessary because estimating a function from a finite noisy sample is an ill-posed inverse problem; the larger \\( \\lambda \\) is, the smoother the estimator.[9](https://ar5iv.labs.arxiv.org/html/1807.02582)\n\n## How it is done\n\nThe practitioner's workflow has three decisions. First, choose a kernel and its hyperparameters; published experiments use radial basis function, Laplacian, and Matérn-5/2 kernels,[2](https://www.jmlr.org/papers/v27/25-0385.html) and the Gaussian kernel is common in molecular property prediction.[11](https://mrupp.info/Data/2019strkghr_jchemphys.pdf)\n\nSecond, choose the regularization parameter \\( \\lambda \\) by cross-validation or leave-one-out estimates.[12](https://teazrq.github.io/stat542/notes/KernelRidge.pdf) • [3](https://web2.qatar.cmu.edu/~gdicaro/10315-Fall19/additional/welling-notes-on-kernel-ridge.pdf) KRR has a closed-form leave-one-out score, \\( g_{\\mathrm{LOO}}(\\lambda) = \\|H^{-1} \\cdot (y - K \\cdot \\alpha)\\|^{2} \\) with \\( H = I_{n} - \\mathrm{diag}(K \\cdot (K + \\lambda I_{n})^{-1}) \\), though computing it naively requires the inverse of \\( (K + \\lambda I_{n}) \\) at \\( O(n^{3}) \\) cost.[13](https://www.ms.k.u-tokyo.ac.jp/sugi/2009/LargeScaleKernel.pdf)\n\nThird, solve the linear system \\( (K + \\lambda I_{n}) \\cdot \\alpha = y \\), typically by Cholesky factorization with forward-backward substitution at \\( O(n^{3}) \\) cost, or by conjugate gradients at \\( O(r \\cdot n^{2}) \\) for r iterations.[13](https://www.ms.k.u-tokyo.ac.jp/sugi/2009/LargeScaleKernel.pdf) All training samples must be stored, since prediction evaluates the kernel against every training point.[12](https://teazrq.github.io/stat542/notes/KernelRidge.pdf)\n\n## Origin\n\nThe ridge estimator that KRR regularizes with was introduced by Arthur E. Hoerl and Robert W. Kennard in \"Ridge Regression: Biased Estimation for Nonorthogonal Problems\" (Technometrics, 1970); the 1970 paper notes that A. E. Hoerl first suggested the idea in 1962 to control the instability of least-squares estimates when prediction vectors are nonorthogonal, and gives the closed form \\( (X' \\cdot X + k \\cdot I)^{-1} \\cdot X' \\cdot Y \\) with \\( k \\geq 0 \\).[14](https://doi.org/10.1080/00401706.1970.10488634) • [15](https://homepages.math.uic.edu/~lreyzin/papers/ridge.pdf)\n\nThe kernelized form was reported in a paper describing a dual version of least-squares and ridge regression that allows the use of kernel functions, applied to ANOVA-enhanced infinite-node splines and evaluated on the Boston Housing data set; that paper notes the approach is closely related to Vapnik's kernel method as used in the Support Vector Machine.[10](https://media.wix.com/ugd/5256f1_27f0134dd65d4b528c5833b054ba2669.pdf) A historical review of kernel methods also records a parallel statistical line in which positive definite kernels were used for time series analysis and for regression estimation and the solution of inverse problems.[16](https://www-ai.cs.tu-dortmund.de/de/LEHRE/FACHPROJEKT/SS14/Papers/kernelmethods.pdf) Interest in kernel methods was later revived by the neural tangent kernel, introduced by Arthur Jacot, Franck Gabriel, and Clément Hongler in 2018 (HAL), although NTKs are computationally complex and often require large computing clusters due to memory constraints.[17](https://arxiv.org/pdf/2301.11414v2.pdf)\n\n## Variants\n\nBecause the exact solver scales as \\( O(n^{3}) \\) in time and \\( O(n^{2}) \\) in memory,[4](https://ar5iv.labs.arxiv.org/html/2509.02649) • [18](https://web.stanford.edu/~pilanci/papers/YangPilWai17.pdf) several approximation families trade accuracy for cost:\n\n- Nyström methods approximate the kernel matrix from m sampled columns, computing a square root of K in \\( O(m^{2} \\cdot n) \\) time, which is linear in n for fixed m.[19](https://www.di.ens.fr/~fbach/learning_theory_class/lecture6.pdf)\n- Random features approximate kernels of the form \\( k(x,x') = \\int \\phi(x,v) \\cdot \\phi(x',v)\\,d\\mu(v) \\) by an empirical average over sampled \\( v_{i} \\), giving an explicit feature map with \\( m \\ll n \\) features.[19](https://www.di.ens.fr/~fbach/learning_theory_class/lecture6.pdf) Nyström methods and random feature expansions reduce the cost to \\( O(n^{3/2}) \\) but introduce additional error terms, often degrading empirical performance.[4](https://ar5iv.labs.arxiv.org/html/2509.02649)\n- Randomized sketches approximate KRR from m-dimensional sketches of the kernel matrix with optimality guarantees.[18](https://web.stanford.edu/~pilanci/papers/YangPilWai17.pdf)\n- Reduced-rank KRR (RRKRR) produces an optimally sparse kernel expansion functionally identical to conventional KRR, outperforming sparse least-squares SVMs on the [Motorcycle](https://www.edgepedia.org/motorcycle) and Boston Housing benchmarks, and stores at most \\( |S| \\) columns of the kernel matrix.[5](http://theoval.cmp.uea.ac.uk/publications/pdf/npl2002a.pdf)\n- Inducing-points methods such as Falkon solve restricted problems with k centers; they are compared as baselines in recent large-scale solver evaluations.[2](https://www.jmlr.org/papers/v27/25-0385.html)\n\nRecent work attacks the \\( O(n^{3}) \\) bottleneck directly. ASkotch, a scalable accelerated iterative solver for full KRR built on sketch-and-project methods and Nyström approximations, provably obtains linear convergence without condition-number assumptions and outperforms EigenPro 2.0, EigenPro 3.0, PCG, and Falkon on 23 large-scale KRR problems, typically with \\( n \\geq 10^{5} \\).[2](https://www.jmlr.org/papers/v27/25-0385.html) Randomized preconditioners accelerate conjugate gradients: RPCholesky preconditioning solves full-data KRR in \\( O(n^{2}) \\) operations for fixed accuracy under sufficiently rapid eigenvalue decay, and KRILL preconditioning for restricted KRR with \\( k \\ll n \\) centers costs \\( O((n+k^{2}) \\cdot k\\log k) \\) operations with no eigenvalue-decay assumption, targeting \\( 10^{4} \\leq n \\leq 10^{7} \\).[20](https://link.springer.com/article/10.1007/s10444-026-10360-1) Kernel gradient descent (KGD) solves KRR in \\( O(T \\cdot n^{2}) \\) operations for T iterations, cheaper than the \\( O(n^{3}) \\) closed-form solve when \\( T < n \\).[6](https://arxiv.org/html/2311.01762v2) A Fourier/NUFFT-based framework achieves exact kernel ridge regression with \\( O(n\\log n) \\) time and memory, GPU-accelerated.[4](https://ar5iv.labs.arxiv.org/html/2509.02649)\n\n## Applications\n\nKRR is listed among the popular choices for quantum-chemistry machine learning (QM/ML), where applying the kernel trick to ridge regression yields the regression coefficients \\( \\alpha \\) in closed form.[21](https://pure.mpg.de/rest/items/item_2175859/component/file_2186290/content) A typical use models the relationship between molecular structures and HOMO energies, mapping training samples into a high-dimensional space via a kernel function such as the Gaussian kernel.[11](https://mrupp.info/Data/2019strkghr_jchemphys.pdf) KRR is also applied in healthcare and, more recently, scientific machine learning.[2](https://www.jmlr.org/papers/v27/25-0385.html) In clinical statistics, KRR is a regression method for pattern recognition in high-dimensional clinical data.[22](https://link.springer.com/book/10.1007/978-3-031-10717-7) In genomic prediction, RKHS regression (the KRR-type estimator) has been applied to rice traits, with cross-validation runtimes that are trait- and setup-dependent.[23](https://www.frontiersin.org/journals/genetics/articles/10.3389/fgene.2016.00145/full)\n\n## Limitations and alternatives\n\nThe dominant limitation is scale: solving the n × n system costs \\( O(n^{3}) \\) in time and \\( O(n^{2}) \\) in memory,[4](https://ar5iv.labs.arxiv.org/html/2509.02649) and the standard direct Cholesky method in practice limits KRR to about \\( n \\leq 10^{4} \\) data points.[20](https://link.springer.com/article/10.1007/s10444-026-10360-1) The problem is also ill-conditioned because the kernel matrix often has very small eigenvalues.[19](https://www.di.ens.fr/~fbach/learning_theory_class/lecture6.pdf) Too small a \\( \\lambda \\) invites overfitting; regularization controls the smoothness of the estimator.[9](https://ar5iv.labs.arxiv.org/html/1807.02582)\n\nCompared with support vector regression, KRR uses squared error loss while SVR uses \\( \\epsilon \\)-insensitive loss, both with L2 regularization. KRR fitting is closed-form and typically faster for medium-sized datasets, but its non-sparse model, which has no concept of support vectors,[3](https://web2.qatar.cmu.edu/~gdicaro/10315-Fall19/additional/welling-notes-on-kernel-ridge.pdf) is slower at prediction time. In scikit-learn's benchmark on a noisy sinusoidal dataset, fitting KRR was approximately seven times faster than fitting SVR with grid search, while SVR, which used roughly one third of 100 training points as support vectors, was faster at predicting 100,000 targets; KRR fitting is faster for training sets under about 1000 samples, while SVR scales better beyond that.[1](https://scikit-learn.org/stable/modules/kernel_ridge.html)\n\nCompared with [Gaussian process regression](https://www.edgepedia.org/gaussian-process-regression), KRR and GPR produce close predictive results, but GPR additionally outputs uncertainty (standard deviation or covariance) at higher prediction-time cost.[7](https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_compare_gpr_krr.html) The KRR estimate coincides with the posterior mean of kriging, or Gaussian process regression.[6](https://arxiv.org/html/2311.01762v2)\n\n## References\n\n1. [1.3. Kernel ridge regression, scikit-learn documentation](https://scikit-learn.org/stable/modules/kernel_ridge.html)\n2. [Have ASkotch: A Neat Solution for Large-Scale Kernel Ridge Regression](https://www.jmlr.org/papers/v27/25-0385.html)\n3. [Notes on Kernel Ridge Regression (Max Welling)](https://web2.qatar.cmu.edu/~gdicaro/10315-Fall19/additional/welling-notes-on-kernel-ridge.pdf)\n4. [Fast kernel methods: Sobolev, physics-Informed, and additive models](https://ar5iv.labs.arxiv.org/html/2509.02649)\n5. [Reduced Rank Kernel Ridge Regression](http://theoval.cmp.uea.ac.uk/publications/pdf/npl2002a.pdf)\n6. [Solving Kernel Ridge Regression with Gradient Descent for a Non-Constant Kernel](https://arxiv.org/html/2311.01762v2)\n7. [Comparison of kernel ridge and Gaussian process regression, scikit-learn example](https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_compare_gpr_krr.html)\n8. [Kernel ridge regression (book chapter)](https://mlweb.loria.fr/book/en/kernelridgeregression.html)\n9. [Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences](https://ar5iv.labs.arxiv.org/html/1807.02582)\n10. [Dual Ridge Regression with kernels and ANOVA splines (Boston Housing experiments)](https://media.wix.com/ugd/5256f1_27f0134dd65d4b528c5833b054ba2669.pdf)\n11. [Chemical diversity in molecular orbital energy predictions with kernel ridge regression](https://mrupp.info/Data/2019strkghr_jchemphys.pdf)\n12. [STAT 542: Statistical Learning, RKHS and Kernel Ridge Regression](https://teazrq.github.io/stat542/notes/KernelRidge.pdf)\n13. [Recent Advances and Trends in Large-scale Kernel Methods](https://www.ms.k.u-tokyo.ac.jp/sugi/2009/LargeScaleKernel.pdf)\n14. [Arthur E. Hoerl, Robert W. Kennard (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics.](https://doi.org/10.1080/00401706.1970.10488634)\n15. [Ridge Regression: Biased Estimation for Nonorthogonal Problems (Hoerl & Kennard)](https://homepages.math.uic.edu/~lreyzin/papers/ridge.pdf)\n16. [Kernel Methods in Machine Learning (Hofmann, Schölkopf, Smola)](https://www-ai.cs.tu-dortmund.de/de/LEHRE/FACHPROJEKT/SS14/Papers/kernelmethods.pdf)\n17. [A Simple Algorithm For Scaling Up Kernel Methods](https://arxiv.org/pdf/2301.11414v2.pdf)\n18. [Randomized sketches for kernels: Fast and optimal nonparametric regression](https://web.stanford.edu/~pilanci/papers/YangPilWai17.pdf)\n19. [Learning theory from first principles, lecture 6](https://www.di.ens.fr/~fbach/learning_theory_class/lecture6.pdf)\n20. [Robust, randomized preconditioning for kernel ridge regression](https://link.springer.com/article/10.1007/s10444-026-10360-1)\n21. [Machine learning of molecular properties (QM/ML review, Max Planck repository copy)](https://pure.mpg.de/rest/items/item_2175859/component/file_2186290/content)\n22. [Kernel Ridge Regression in Clinical Research](https://link.springer.com/book/10.1007/978-3-031-10717-7)\n23. [A Unified and Comprehensible View of Parametric and Kernel Methods for Genomic Prediction with Application to Rice](https://www.frontiersin.org/journals/genetics/articles/10.3389/fgene.2016.00145/full)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods › Supervised, unsupervised, and semi-supervised learning › Kernel methods and support vector machines*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://web2.qatar.cmu.edu/~gdicaro/10315-Fall19/additional/welling-notes-on-kernel-ridge.pdf", "https://homepages.math.uic.edu/~lreyzin/papers/ridge.pdf", "https://web.stanford.edu/~pilanci/papers/YangPilWai17.pdf" ], "url": "https://www.edgepedia.org/kernel-ridge-regression", "markdown_url": "https://www.edgepedia.org/kernel-ridge-regression.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Kernel ridge regression\", Edgepedia (EdgeChat), https://www.edgepedia.org/kernel-ridge-regression. Edgepedia Community License 1.0.", "credit_md": "\"[Kernel ridge regression](https://www.edgepedia.org/kernel-ridge-regression)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/kernel-ridge-regression](https://www.edgepedia.org/kernel-ridge-regression). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Kernel ridge regression\", Edgepedia (EdgeChat), https://www.edgepedia.org/kernel-ridge-regression. Edgepedia Community License 1.0.", "speakable": "Kernel ridge regression is a nonlinear regression method that combines ridge regression with the kernel trick, yielding a closed-form predictor used in molecular property prediction and genomic prediction." }