{ "id": "epea4bxrkw", "slug": "bayesian-estimator", "title": "Bayesian estimator", "updated": "2026-10-15", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics" }, { "id": "physical.math.statistics.stats_bayesian", "label": "Bayesian statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_bayesian" } ], "geo": [ { "id": "geo.nongeo.t1000.physical", "label": "Non-geographic · 1000 to 1799: Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1000.physical", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1000", "label": "Non-geographic · 1000 to 1799", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1000" }, { "id": "geo.nongeo.t1000.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1000.physical" } ] }, { "id": "geo.nongeo.t1800.physical.math.statistics", "label": "Non-geographic · 1800 to 1945: Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1800", "label": "Non-geographic · 1800 to 1945", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800" }, { "id": "geo.nongeo.t1800.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical" }, { "id": "geo.nongeo.t1800.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math" }, { "id": "geo.nongeo.t1800.physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics" } ] } ], "excerpt": "A Bayesian estimator is a statistical method that combines a prior distribution with observed data via Bayes' theorem, yielding a posterior distribution from which point estimates and credible intervals are drawn.", "snippet": "A Bayesian estimator is a statistical method that combines a prior distribution with observed data via Bayes' theorem, yielding a posterior distribution from which point estimates and credible intervals are drawn.", "node": "physical.math.statistics.stats_bayesian", "markdown": "# Bayesian estimator\n\nA Bayesian estimator is a statistical method that estimates an unknown parameter by combining a prior distribution with observed data through [Bayes' theorem](https://www.edgepedia.org/bayes-theorem), producing a full posterior distribution from which point estimates, credible intervals, or other summaries are drawn.\n\n| Key fact | Detail |\n|---|---|\n| Core principle | Posterior ∝ prior × likelihood: \\( f_{\\Theta\\mid X}(\\theta\\mid x) \\propto f_{X\\mid\\Theta}(x\\mid\\theta) \\times f_{\\Theta}(\\theta) \\) [1](https://www.artowen.su.domains/courses/200/lec06.pdf) |\n| Outputs | A full posterior distribution; the mean, median, or mode can each serve as a point estimate, and credible intervals summarize spread [1](https://www.artowen.su.domains/courses/200/lec06.pdf) • [2](https://jrnold.github.io/bayesian_notes/estimation-1.html) |\n| Loss-function optimality | Posterior mean under quadratic loss, posterior median under absolute loss, posterior mode (MAP) for discrete parameters under zero-one loss, or as a density maximizer motivated by a small-neighborhood loss or limiting argument for continuous parameters [3](https://link.springer.com/chapter/10.1007/978-3-030-83640-5_1) |\n| Relation to maximum likelihood | With a flat prior, the posterior has the same shape as the likelihood, so the MAP equals the maximum likelihood estimate [4](https://gwern.net/doc/statistics/bayes/2003-brooks.pdf) |\n| Coverage caveat | The frequentist coverage of a Bayesian \\( 1-\\alpha \\) credible set is typically not \\( 1-\\alpha \\) for all parameter values [5](https://www.stat.berkeley.edu/~stark/Preprints/freqBayes09.pdf) |\n| Main failure mode | Bayes estimates can be sensitive to both the prior and the form of the likelihood [6](https://www.annualreviews.org/content/journals/10.1146/annurev-economics-111809-125134) |\n\n## How it works\n\nBayes' theorem converts a prior belief about a parameter into a posterior belief once data are observed. Writing the prior density as \\( f_{\\Theta}(\\theta) \\) and the likelihood of the data as \\( f_{X\\mid\\Theta}(x\\mid\\theta) \\), the posterior satisfies \\( f_{\\Theta\\mid X}(\\theta\\mid x) \\propto f_{X\\mid\\Theta}(x\\mid\\theta) \\times f_{\\Theta}(\\theta) \\).[1](https://www.artowen.su.domains/courses/200/lec06.pdf)\n\nA point estimate is then a summary of this distribution, and Bayes estimates are, by definition, those that minimize the expected posterior loss.[7](https://www.uv.es/~bernardo/2007Sort.pdf) For a loss function \\( \\rho \\), the Bayes \\( \\rho \\)-risk for a prior \\( \\pi \\) is the smallest average \\( \\rho \\)-risk of any estimator for that prior, and an estimator attaining it is a [Bayes estimator](https://www.edgepedia.org/bayes-estimator); under mean squared error risk the Bayes estimator is the mean of the marginal posterior distribution given the data.[5](https://www.stat.berkeley.edu/~stark/Preprints/freqBayes09.pdf) Under the quadratic loss \\( \\ell(\\theta,\\delta)=(\\theta-\\delta)^2 \\) the Bayes estimate is the posterior mean \\( \\delta^{\\pi}(x_{1:n}) = E_{\\pi}(\\theta\\mid x_{1:n}) \\), which is also the minimum mean squared error estimator.[3](https://link.springer.com/chapter/10.1007/978-3-030-83640-5_1) • [8](https://www.cs.cmu.edu/~harchol/Probability/chapters/chpt17.pdf) Under absolute loss \\( \\ell(\\theta,\\delta)=|\\theta-\\delta| \\) it is the posterior median; under zero-one loss it is the posterior mode, the maximum a posteriori (MAP) estimate; and under a linear loss with asymmetric penalties \\( c_1 \\) and \\( c_2 \\) it is the \\( c_2/(c_1+c_2) \\)-th posterior quantile.[3](https://link.springer.com/chapter/10.1007/978-3-030-83640-5_1) • [2](https://jrnold.github.io/bayesian_notes/estimation-1.html)\n\nThe connection to maximum likelihood is exact in a limiting case: with a flat prior the posterior has the same shape as the likelihood \\( f(x\\mid\\theta) \\), so the posterior mode coincides with the maximum likelihood estimate.[4](https://gwern.net/doc/statistics/bayes/2003-brooks.pdf) Bayes estimates can also approximate maximum likelihood estimates when a suitable transformation makes the posterior close to a normal distribution, which is useful when the amount of data is small.[9](https://esj-journals.onlinelibrary.wiley.com/doi/10.1007/s10144-015-0526-x)\n\n## How it is done\n\nThe practitioner's workflow has four steps. First, choose a likelihood that describes how the data arise given the parameter. Second, choose a prior. If the posterior lands in the same distributional family as the prior, the prior is said to be conjugate to the likelihood, and the posterior is available in closed form.[10](https://www2.stat.duke.edu/courses/Spring23/sta732.01/Lecture10.pdf) Third, compute the posterior, either analytically under conjugacy or numerically. When inversion or integration is intractable, credible intervals may require numerical algorithms.[3](https://link.springer.com/chapter/10.1007/978-3-030-83640-5_1) In hierarchical models the posterior typically has no closed form, so the standard strategy is to set up a [Markov chain](https://www.edgepedia.org/markov-chain) with the posterior as its stationary distribution and run it long enough to obtain approximate posterior samples, the [Markov chain Monte Carlo](https://www.edgepedia.org/markov-chain-monte-carlo) (MCMC) approach.[11](https://www2.stat.duke.edu/courses/Spring23/sta732.01/Lecture11.pdf) Fourth, summarize: report a posterior mean, median, or mode, and an interval.\n\nCredible intervals are not unique. Common choices are equal-tailed intervals and highest posterior density (HPD) intervals, credible sets of minimum posterior volume; in one dimension with a unimodal posterior the HPD set can be represented as a shortest credible interval, and when the posterior is not unimodal and symmetric, the HPD interval differs from the equal-tailed one.[2](https://jrnold.github.io/bayesian_notes/estimation-1.html) For large models where MCMC is too slow, variational inference forms a Gaussian approximation \\( p(\\theta\\mid y_{1:N}) \\approx q_{\\phi}(\\theta) = N(\\theta\\mid\\mu,\\Sigma) \\) and learns \\( \\phi=(\\mu,\\Sigma) \\) by minimizing \\( \\mathrm{KL}[q_{\\phi}(\\theta)\\,\\|\\,p(\\theta\\mid y_{1:N})] \\), with gradients approximated by [Monte Carlo](https://www.edgepedia.org/monte-carlo) and reparameterization.[12](https://arxiv.org/abs/2406.00104)\n\n## Origin\n\nA Bayesian calculation can be framed as a [thought experiment](https://www.edgepedia.org/thought-experiment) that introduces the uniform prior.[13](http://www.stat.ucla.edu/~ywu/Bayesian.pdf) Both Bayes and Laplace knew the relation now called Bayes' theorem, and both were \"objective Bayesians\" who viewed the prior with suspicion and often used a flat prior.[14](https://people.eecs.berkeley.edu/%7Ejordan/courses/260-spring10/lectures/lecture1.pdf) Works by de Laplace, de Finetti, and Jeffreys count among the seminal books of [Bayesian statistics](https://www.edgepedia.org/bayesian-statistics).[15](https://www.eolss.net/sample-chapters/c02/E6-02-04-06.pdf)\n\nThe historian A. I. Dale suggests the rigorous development of Bayes's work into a statistical tool can be viewed as a reaction to Fisher's evolution of sampling theory, with the 1930s also seeing the birth of the Neyman–Pearson and later Wald approaches.[16](https://unina2.on-line.it/sebina/repository/catalogazione/documenti/Dale%20-%20A%20history%20of%20inverse%20probability%20from%20Thomas%20Bayes%20to%20Karl%20Pearson.pdf) The approach re-emerged in the late 1980s, driven by rapid developments in computing and by the desire to model increasingly complex scientific phenomena that older sampling theories addressed poorly.[4](https://gwern.net/doc/statistics/bayes/2003-brooks.pdf)\n\n## Variants\n\n**Empirical Bayes** estimates the prior itself from the data. The nonparametric formulation, which leaves the prior completely unspecified, was first set out by [Herbert Robbins](https://www.edgepedia.org/herbert-robbins) in \"An Empirical Bayes Approach to Statistics,\" Proceedings of the Third Berkeley Symposium on Mathematical Statistics and [Probability](https://www.edgepedia.org/probability).[17](https://biostat.jhsph.edu/%7Efdominic/teaching/bio656/labs/labs09/Casella.EmpBayes.pdf) The parametric approach instead specifies a family of prior distributions; Carl N. Morris gave its theory in 1983 in the [Journal of the American Statistical Association](https://www.edgepedia.org/journal-of-the-american-statistical-association), showing that its compromise estimator is \"Stein's celebrated estimator\" (James and Stein 1961).[18](https://doi.org/10.1080/01621459.1983.10477920) If \\( X_i \\mid \\theta_i \\sim N(\\theta_i, 1) \\) and the prior is normal, \\( \\theta_i \\sim N(\\mu, \\tau^2) \\), the Bayes estimate is the posterior mean \\( b_1^*(X_i) = \\mu + (1 - (1+\\tau^2)^{-1}) \\cdot (X_i - \\mu) \\), and in the empirical Bayes setting \\( \\tau^2 \\) is estimated from the marginal distribution of the \\( X_i \\).[19](https://www.jhanley.biostat.mcgill.ca/bios602/MultilevelData/EfronMorrisJASA1975.pdf)\n\n**Hierarchical Bayes** addresses prior choice by treating the prior's parameters as hyperparameters estimated jointly with the data model; Gelman's history notes it propagates uncertainty in the hyperparameters and allows parameters to vary over time as well as across groups.[20](https://sites.stat.columbia.edu/gelman/research/published/bayes_history.pdf)\n\n**Variational inference** replaces sampling with optimization and is surveyed for statisticians by Blei, Kucukelbir, and McAuliffe in 2017 in the Journal of the American Statistical Association.[21](https://doi.org/10.1080/01621459.2017.1285773)\n\n## Applications\n\nIn Efron and Morris's applications, the mean squared error of Stein-type rules was less than half that of the sample mean [19](https://www.jhanley.biostat.mcgill.ca/bios602/MultilevelData/EfronMorrisJASA1975.pdf), and empirical Bayes confidence intervals for the equal-variance normal case can be shorter than standard intervals for every \\( \\theta_i \\).[18](https://doi.org/10.1080/01621459.1983.10477920) In machine learning, the posteriors package implements stochastic gradient MCMC, variational inference, and Laplace approximation for large PyTorch and [Hugging Face](https://www.edgepedia.org/hugging-face) models [12](https://arxiv.org/abs/2406.00104), and Generative Bayesian Computation trains a deep neural network to map samples from a base distribution to the posterior, avoiding densities entirely, with applications to likelihood-free and parametric inference, prediction, and maximum expected utility analysis.[22](https://www.mdpi.com/1099-4300/27/7/683)\n\n## Limitations and alternatives\n\nThe main difficulty is prior choice, especially in high dimension, and a frequentist may doubt whether the prior puts any mass near the true parameter.[10](https://www2.stat.duke.edu/courses/Spring23/sta732.01/Lecture10.pdf) Bayes estimates can be sensitive to both the prior and the form of the likelihood, although non-Bayesian analyses also incorporate prior information.[6](https://www.annualreviews.org/content/journals/10.1146/annurev-economics-111809-125134) Model misspecification is a second failure mode: [Bayesian inference](https://www.edgepedia.org/bayesian-inference) with a flawed model can produce unreliable conclusions, and in most cases a conventional Bayesian analysis is not meaningful under serious misspecification. Three classes of remedies are discussed in the literature: restricted likelihood methods using a non-sufficient summary of the data, modular inference built from coupled submodels of which some are correctly specified, and the use of a reference model.[23](https://arxiv.org/html/2305.08429v2)\n\nCredible intervals also differ in interpretation from confidence intervals. A 95% credible interval is read directly as a 95% probability that \\( \\theta \\) lies in the interval, which is a strength of the Bayesian construction.[24](https://bookdown.org/johnnydoorn/bayesbookdown/how-do-models-estimate.html) The frequentist coverage of such a set, however, is typically not \\( 1-\\alpha \\) for all parameter values [5](https://www.stat.berkeley.edu/~stark/Preprints/freqBayes09.pdf), and typically no estimator has smallest risk for all possible states of the world.[5](https://www.stat.berkeley.edu/~stark/Preprints/freqBayes09.pdf)\n\n## References\n\n1. [Lecture 6: Bayesian estimation (Art Owen, Stanford)](https://www.artowen.su.domains/courses/200/lec06.pdf)\n2. [Estimation | Updating: A Set of Bayesian Notes (Jeffrey Arnold)](https://jrnold.github.io/bayesian_notes/estimation-1.html)\n3. [Introduction to Bayesian Statistical Inference (Springer chapter)](https://link.springer.com/chapter/10.1007/978-3-030-83640-5_1)\n4. [Bayesian computation: a statistical revolution (S. P. Brooks, 2003)](https://gwern.net/doc/statistics/bayes/2003-brooks.pdf)\n5. [Bayesian Inference in Inverse Problems (Stark)](https://www.stat.berkeley.edu/~stark/Preprints/freqBayes09.pdf)\n6. [Confronting Prior Convictions: On Issues of Prior Sensitivity and Likelihood Robustness in Bayesian Analysis](https://www.annualreviews.org/content/journals/10.1146/annurev-economics-111809-125134)\n7. [Objective Bayesian point and region estimation (Bernardo, SORT 2007)](https://www.uv.es/~bernardo/2007Sort.pdf)\n8. [Bayesian Statistical Inference (CMU, Ch. 17 of Harchol-Balter's probability text)](https://www.cs.cmu.edu/~harchol/Probability/chapters/chpt17.pdf)\n9. [Bayes estimates as an approximation to maximum likelihood estimates (Ecological Research)](https://esj-journals.onlinelibrary.wiley.com/doi/10.1007/s10144-015-0526-x)\n10. [STA732 Lecture 10: Bayes pros and cons (Duke)](https://www2.stat.duke.edu/courses/Spring23/sta732.01/Lecture10.pdf)\n11. [STA732 Lecture 11: Empirical Bayes and Hierarchical Bayes (Duke)](https://www2.stat.duke.edu/courses/Spring23/sta732.01/Lecture11.pdf)\n12. [Scalable Bayesian Learning with posteriors](https://arxiv.org/abs/2406.00104)\n13. [A Bayesian Way of Thinking (Ying Nian Wu, UCLA)](http://www.stat.ucla.edu/~ywu/Bayesian.pdf)\n14. [Course lecture notes on Bayesian history (M. I. Jordan, UC Berkeley)](https://people.eecs.berkeley.edu/%7Ejordan/courses/260-spring10/lectures/lecture1.pdf)\n15. [Bayesian Statistics (EOLSS encyclopedia chapter)](https://www.eolss.net/sample-chapters/c02/E6-02-04-06.pdf)\n16. [A History of Inverse Probability from Thomas Bayes to Karl Pearson (A. I. Dale)](https://unina2.on-line.it/sebina/repository/catalogazione/documenti/Dale%20-%20A%20history%20of%20inverse%20probability%20from%20Thomas%20Bayes%20to%20Karl%20Pearson.pdf)\n17. [An Introduction to Empirical Bayes Data Analysis (Casella)](https://biostat.jhsph.edu/%7Efdominic/teaching/bio656/labs/labs09/Casella.EmpBayes.pdf)\n18. [Carl N. Morris (1983). Parametric Empirical Bayes Inference: Theory and Applications. Journal of the American Statistical Association.](https://doi.org/10.1080/01621459.1983.10477920)\n19. [Data Analysis Using Stein's Estimator and its Generalizations (Efron & Morris, JASA 1975)](https://www.jhanley.biostat.mcgill.ca/bios602/MultilevelData/EfronMorrisJASA1975.pdf)\n20. [The history of statistics in 1933 (or, a history of Bayesian statistics), A. Gelman](https://sites.stat.columbia.edu/gelman/research/published/bayes_history.pdf)\n21. [David M. Blei, Alp Kucukelbir, Jon D. McAuliffe (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association.](https://doi.org/10.1080/01621459.2017.1285773)\n22. [Generative AI for Bayesian Computation (Entropy, 2025)](https://www.mdpi.com/1099-4300/27/7/683)\n23. [Bayesian inference for misspecified generative models](https://arxiv.org/html/2305.08429v2)\n24. [A Brief Introduction to Bayesian Inference (Doorn, bookdown)](https://bookdown.org/johnnydoorn/bayesbookdown/how-do-models-estimate.html)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Bayesian statistics*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Oct 15, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://www.stat.berkeley.edu/~stark/Preprints/freqBayes09.pdf", "https://www.cs.cmu.edu/~harchol/Probability/chapters/chpt17.pdf", "http://www.stat.ucla.edu/~ywu/Bayesian.pdf" ], "url": "https://www.edgepedia.org/bayesian-estimator", "markdown_url": "https://www.edgepedia.org/bayesian-estimator.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Bayesian estimator\", Edgepedia (EdgeChat), https://www.edgepedia.org/bayesian-estimator. Edgepedia Community License 1.0.", "credit_md": "\"[Bayesian estimator](https://www.edgepedia.org/bayesian-estimator)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/bayesian-estimator](https://www.edgepedia.org/bayesian-estimator). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Bayesian estimator\", Edgepedia (EdgeChat), https://www.edgepedia.org/bayesian-estimator. Edgepedia Community License 1.0.", "speakable": "A Bayesian estimator is a statistical method that combines a prior distribution with observed data via Bayes' theorem, yielding a posterior distribution from which point estimates and credible intervals are drawn." }