{ "id": "ep6dh5aga5", "slug": "bernoulli-model", "title": "Bernoulli model", "updated": "2026-09-30", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics" }, { "id": "physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference" } ], "geo": [ { "id": "geo.nongeo.t1000.physical", "label": "Non-geographic · 1000 to 1799: Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1000.physical", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1000", "label": "Non-geographic · 1000 to 1799", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1000" }, { "id": "geo.nongeo.t1000.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1000.physical" } ] }, { "id": "geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference", "label": "Non-geographic · 1800 to 1945: Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1800", "label": "Non-geographic · 1800 to 1945", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800" }, { "id": "geo.nongeo.t1800.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical" }, { "id": "geo.nongeo.t1800.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math" }, { "id": "geo.nongeo.t1800.physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics" }, { "id": "geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1800.physical.math.statistics.stats_statistical_inference" } ] } ], "excerpt": "The Bernoulli model is a statistical model for binary data where each observation is a success or failure with fixed success probability, underlying binomial and logistic models.", "snippet": "The Bernoulli model is a statistical model for binary data where each observation is a success or failure with fixed success probability, underlying binomial and logistic models.", "node": "physical.math.statistics.stats_statistical_inference", "markdown": "# Bernoulli model\n\nThe Bernoulli model is a statistical model for binary data in which each observation takes one of two outcomes, conventionally success or failure, with a fixed success probability. Its probability mass function is \\( f(y) = \\pi^{y} \\cdot (1-\\pi)^{1-y} \\) for \\( y \\in \\{0,1\\} \\), where the single parameter \\( \\pi \\) is the probability of success and also the mean of the distribution.[1](https://stats.libretexts.org/Courses/Knox_College/Linear_Models_and_Rurita_Kralovstvi/15%3A_Binary_Dependent_Variables/15.03%3A_The_Mathematics) Because one parameter fully determines both the mean (\\( \\pi \\)) and the variance (\\( \\pi \\cdot (1-\\pi) \\)), the model is the elementary building block for dichotomous data and for the binomial, logistic-regression, and Naive Bayes models built on it.[2](https://www.mdpi.com/2571-905X/7/1/16) A single [Bernoulli trial](https://www.edgepedia.org/bernoulli-trial) with success probability \\( \\pi \\) is equivalently a Binomial\\([1, \\pi]\\) variable.[3](http://www.gllamm.org/JEBSredundant_07.pdf)\n\n| Key fact | Value or statement |\n|---|---|\n| Probability mass function | \\( f(y) = \\pi^{y} \\cdot (1-\\pi)^{1-y} \\), \\( y \\in \\{0,1\\} \\)[1](https://stats.libretexts.org/Courses/Knox_College/Linear_Models_and_Rurita_Kralovstvi/15%3A_Binary_Dependent_Variables/15.03%3A_The_Mathematics) |\n| Mean and variance | \\( E[X] = \\pi \\), \\( \\mathrm{Var}[X] = \\pi \\cdot (1-\\pi) \\)[4](https://www.ccs.neu.edu/home/vip/teach/MLcourse/3_generative_models/lecture_notes/Murphy_bernoulli_multinom.pdf) |\n| Maximum likelihood estimator | \\( \\hat{\\pi} = Y/n \\), the sample proportion, with \\( V(\\hat{\\pi}) = \\pi \\cdot (1-\\pi)/n \\)[5](https://online.stat.psu.edu/stat504/Lesson02) |\n| Relation to binomial | One Bernoulli trial is Binomial\\([1, \\pi]\\); \\( n \\) independent trials give Binomial\\((n, \\pi)\\)[3](http://www.gllamm.org/JEBSredundant_07.pdf) |\n| Historical origin | The Law of Large Numbers for Bernoulli experiments appeared in *Ars Conjectandi*[6](https://arxiv.org/pdf/1309.6488) |\n| Naive Bayes event model | Multinomial Naive Bayes gave on average a 27% reduction in error over the multivariate Bernoulli model across five text corpora[7](http://www.kamalnigam.com/papers/multinomial-aaaiws98.pdf) |\n| Interval caution | The Wald interval's coverage probabilities tend to be too small; the Wilson score interval stays close to nominal even for very small samples[8](https://math.unm.edu/%7Ejames/Agresti1998.pdf) |\n\n## How it works\n\nFor independent Bernoulli observations \\( y_1, \\ldots, y_n \\) with parameter \\( \\pi \\), the likelihood is \\( L(\\pi) = \\prod_{i} f(y_i; \\pi) \\) and the log-likelihood is \\( \\ell(\\pi) = \\left(\\sum_i y_i\\right) \\log \\pi + \\left(n - \\sum_i y_i\\right) \\log(1-\\pi) \\). The likelihood depends on the data only through \\( T = \\sum_i y_i \\), the number of successes, which is therefore sufficient.[9](https://projecteuclid.org/ebook/Download?isFullBook=False&urlid=10.1214%2Fcbms%2F1462106040) Maximizing the log-likelihood gives \\( \\hat{\\pi} = Y/n \\), the sample proportion, with \\( E(\\hat{\\pi}) = \\pi \\) and \\( V(\\hat{\\pi}) = \\pi \\cdot (1-\\pi)/n \\).[4](https://www.ccs.neu.edu/home/vip/teach/MLcourse/3_generative_models/lecture_notes/Murphy_bernoulli_multinom.pdf) • [5](https://online.stat.psu.edu/stat504/Lesson02)\n\nThe variance \\( \\pi \\cdot (1-\\pi) \\) is a quadratic function of the mean, largest at \\( \\pi = 0.5 \\).[1](https://stats.libretexts.org/Courses/Knox_College/Linear_Models_and_Rurita_Kralovstvi/15%3A_Binary_Dependent_Variables/15.03%3A_The_Mathematics) This fixed mean–variance link is what inference exploits: by the central limit theorem, the standardized estimator is approximately a standard normal pivot for \\( \\pi \\).[10](https://stats.libretexts.org/Bookshelves/Probability_Theory/Probability_Mathematical_Statistics_and_Stochastic_Processes_%28Siegrist%29/08%3A_Set_Estimation/8.03%3A_Estimation_in_the_Bernoulli_Model) When \\( n \\) independent trials with constant \\( \\pi \\) are grouped, the success count follows Binomial\\((n, \\pi)\\); the Bernoulli model is the \\( n = 1 \\) case of that frame.[3](http://www.gllamm.org/JEBSredundant_07.pdf)\n\n## How it is done\n\nThree large-sample tests of \\( H_0: \\pi = \\pi_0 \\) are standard. The [Wald test](https://www.edgepedia.org/wald-test) uses \\( Z_w = (\\hat{\\pi} - \\pi_0)/\\sqrt{\\hat{\\pi} \\cdot (1-\\hat{\\pi})/n} \\), approximately standard normal for large \\( n \\); the [score test](https://www.edgepedia.org/score-test) uses \\( Z_s = (\\hat{\\pi} - \\pi_0)/\\sqrt{\\pi_0 \\cdot (1-\\pi_0)/n} \\), substituting the null value in the variance; and the likelihood-ratio statistic is \\( G^2 = 2\\left[\\ell(\\hat{\\pi}) - \\ell(\\pi_0)\\right] = 2\\left(y\\log\\frac{\\hat{\\pi}}{\\pi_0} + (n-y)\\log\\frac{1-\\hat{\\pi}}{1-\\pi_0}\\right) \\).[5](https://online.stat.psu.edu/stat504/Lesson02)\n\nThe Wald interval \\( \\hat{\\pi} \\pm z_{\\alpha/2} \\cdot \\sqrt{\\hat{\\pi} \\cdot (1-\\hat{\\pi})/n} \\) (with \\( z_{.025} = 1.96 \\) for 95% coverage) performs poorly when \\( \\pi \\) is near 0 or 1, even for large \\( n \\), and coverage can be low for moderate \\( n \\) even when \\( \\pi \\) is near 0.5.[5](https://online.stat.psu.edu/stat504/Lesson02) • [11](https://alanagresti.com/articles/ci_proportion.pdf) Inverting the score test with the null standard error gives the Wilson interval, whose coverage is close to nominal even for very small sample sizes; its center is \\( (\\hat{\\pi} + z^2/2n)/(1 + z^2/n) \\).[8](https://math.unm.edu/%7Ejames/Agresti1998.pdf) • [5](https://online.stat.psu.edu/stat504/Lesson02) The Beta distribution is the conjugate prior for the Bernoulli likelihood; \\( \\alpha_1 = \\alpha_0 = 1 \\) is the uniform (Laplace) prior, and \\( \\alpha_1 = \\alpha_0 = 1/2 \\) is the Jeffreys prior, which yields the equal-tailed Jeffreys interval.[4](https://www.ccs.neu.edu/home/vip/teach/MLcourse/3_generative_models/lecture_notes/Murphy_bernoulli_multinom.pdf) • [12](http://www-stat.wharton.upenn.edu/%7Etcai/paper/Binomial-StatSci.pdf) For planning, a conservative sample size for confidence \\( 1-\\alpha \\) and margin of error \\( d \\) can be chosen.[10](https://stats.libretexts.org/Bookshelves/Probability_Theory/Probability_Mathematical_Statistics_and_Stochastic_Processes_%28Siegrist%29/08%3A_Set_Estimation/8.03%3A_Estimation_in_the_Bernoulli_Model)\n\n## Origin\n\nThe multivariate Bernoulli event model for Naive Bayes text classification, together with the comparison against the multinomial event model, is credited to Andrew McCallum and Kamal Nigam in 1998.[7](http://www.kamalnigam.com/papers/multinomial-aaaiws98.pdf) The model's historical roots lie in the *Ars Conjectandi* (1713): the relative frequency of successes in independent repetitions of Bernoulli experiments converges in probability to the unknown success probability as the number of trials increases indefinitely.[13](https://www.sciencedirect.com/science/article/pii/S0315086014000482) • [6](https://arxiv.org/pdf/1309.6488) Bayesian estimation of \\( \\pi \\) solves [Jacob Bernoulli](https://www.edgepedia.org/jacob-bernoulli)'s inversion problem using Bayes's Theorem under a uniform prior on the binomial success probability, taking the posterior mean as the predictive probability, what is now called the [Bayes estimator](https://www.edgepedia.org/bayes-estimator).[6](https://arxiv.org/pdf/1309.6488)\n\n## Variants\n\nIn text classification, the multivariate Bernoulli model represents a document as a binary vector over the vocabulary, each term's occurrence governed by a [Bernoulli distribution](https://www.edgepedia.org/bernoulli-distribution) with class-conditional parameter \\( \\theta_{tk|ci} = P(t_k|c_i; \\theta) \\).[7](http://www.kamalnigam.com/papers/multinomial-aaaiws98.pdf) • [14](https://ceur-ws.org/Vol-835/paper5.pdf) The maximum likelihood estimate is \\( \\hat{\\theta}_{\\mathrm{ML}} = \\tau_{k,i}/m_i \\), the fraction of documents of class \\( c_i \\) containing term \\( t_k \\); the conjugate beta prior gives the smoothed estimate \\( (\\tau_{k,i}+\\alpha)/(m_i+\\alpha+\\beta) \\), with \\( \\alpha = 1, \\beta = 1 \\) called [Laplace smoothing](https://www.edgepedia.org/additive-smoothing), and Jelinek-Mercer smoothing interpolating the ML estimate with a collection language model.[14](https://ceur-ws.org/Vol-835/paper5.pdf)\n\nIn regression settings, the generalized linear model built on the univariate Bernoulli distribution is logistic regression; extending it to a vector of binary responses gives the multivariate Bernoulli logistic model, which supports point estimation, hypothesis tests, confidence intervals, and LASSO variable selection.[15](https://pages.cs.wisc.edu/~sding/paper/mvb.pdf)\n\n## Applications\n\nThe Bernoulli model estimates \\( P(t|c) \\) as the fraction of documents containing the term and uses binary occurrence information while ignoring the number of occurrences.[16](https://nlp.stanford.edu/IR-book/html/htmledition/the-bernoulli-model-1.html) McCallum and Nigam found that the multivariate Bernoulli performs well with small vocabulary sizes but that the multinomial provides on average a 27% reduction in error at any vocabulary size.[7](http://www.kamalnigam.com/papers/multinomial-aaaiws98.pdf) scikit-learn implements this distinction directly: BernoulliNB is designed for binary/boolean features, while MultinomialNB works with occurrence counts.[17](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB)\n\n## Limitations and alternatives\n\nThe model's central restriction is the fixed mean–variance relationship. For a single dichotomous observation the mean–variance relationship is always fixed, so there is no scope for overdispersion or underdispersion at the Bernoulli level; overdispersion arises in grouped data, where the success count's variance exceeds the binomial variance.[3](http://www.gllamm.org/JEBSredundant_07.pdf) On this point the literature disagrees. The prevailing view holds that excess or deficient Bernoulli variation is nonsensical and that overdispersion requires more than one trial.[2](https://www.mdpi.com/2571-905X/7/1/16) The term \"implicitly overdispersed\" is used for binary logit models which, when converted to grouped format, were proven to be overdispersed, and it is noted that with a binary model you cannot immediately tell that it is extra-dispersed from the ungrouped data itself.[18](http://www.highstat.com/Books/BGS/GLMGLMM/pdfs/HILBE-Can_binary_logistic_models_be_overdispersed2Jul2013.pdf)\n\nFor grouped overdispersion, the standard alternative assumes \\( \\pi \\sim \\mathrm{Beta}(\\alpha, \\beta) \\), so that unconditionally the success count follows a [beta-binomial distribution](https://www.edgepedia.org/beta-binomial-distribution).[19](https://pmc.ncbi.nlm.nih.gov/articles/PMC5736152/) This compound of the beta and binomial distributions, derived by Bayes, remained dormant as a vehicle for abnormal binomial variation.[2](https://www.mdpi.com/2571-905X/7/1/16) The Bernoulli model is the only Naive Bayes event model that models absence of terms explicitly; as a result it typically makes many mistakes on long documents.[16](https://nlp.stanford.edu/IR-book/html/htmledition/the-bernoulli-model-1.html) Two further cautions: for Bernoulli variables, independence and uncorrelatedness are equivalent, so zero correlation cannot be taken as evidence of anything weaker than full independence;[15](https://pages.cs.wisc.edu/~sding/paper/mvb.pdf) and applying a Bernoulli likelihood to non-binary data is a documented failure mode, since modeling \\( [0,1] \\)-valued pixel data with a Bernoulli likelihood, whose support is only \\( \\{0,1\\} \\), is a pervasive error in variational autoencoders with profound consequences.[20](https://proceedings.neurips.cc/paper/2019/file/f82798ec8909d23e55679ee26bb26437-Paper.pdf)\n\n## References\n\n1. [15.3: The Mathematics – Statistics LibreTexts (Knox College)](https://stats.libretexts.org/Courses/Knox_College/Linear_Models_and_Rurita_Kralovstvi/15%3A_Binary_Dependent_Variables/15.03%3A_The_Mathematics)\n2. [Comments on the Bernoulli Distribution and Hilbe's Implicit Extra-Dispersion](https://www.mdpi.com/2571-905X/7/1/16)\n3. [Redundant Overdispersion Parameters in Multilevel Models for Categorical Responses](http://www.gllamm.org/JEBSredundant_07.pdf)\n4. [Bernoullis and Binomials (Kevin Murphy, Machine Learning course notes)](https://www.ccs.neu.edu/home/vip/teach/MLcourse/3_generative_models/lecture_notes/Murphy_bernoulli_multinom.pdf)\n5. [2 Binomial and Multinomial Inference – STAT 504 | Analysis of Discrete Data (Penn State)](https://online.stat.psu.edu/stat504/Lesson02)\n6. [A Tricentenary history of the Law of Large Numbers](https://arxiv.org/pdf/1309.6488)\n7. [A Comparison of Event Models for Naive Bayes Text Classification (McCallum & Nigam, AAAI-98 Workshop)](http://www.kamalnigam.com/papers/multinomial-aaaiws98.pdf)\n8. [Approximate is better than 'exact' for interval estimation of binomial proportions (Agresti & Coull, The American Statistician, 1998)](https://math.unm.edu/%7Ejames/Agresti1998.pdf)\n9. [CBMS monograph (Project Euclid, DOI 10.1214/cbms/1462106040)](https://projecteuclid.org/ebook/Download?isFullBook=False&urlid=10.1214%2Fcbms%2F1462106040)\n10. [8.03: Estimation in the Bernoulli Model (stats.libretexts.org)](https://stats.libretexts.org/Bookshelves/Probability_Theory/Probability_Mathematical_Statistics_and_Stochastic_Processes_%28Siegrist%29/08%3A_Set_Estimation/8.03%3A_Estimation_in_the_Bernoulli_Model)\n11. [On Sample Size Guidelines for Teaching Inference about the Binomial Parameter in Introductory Statistics (Agresti & Coull)](https://alanagresti.com/articles/ci_proportion.pdf)\n12. [Interval Estimation for a Binomial Proportion (Brown, Cai, DasGupta, Statistical Science)](http://www-stat.wharton.upenn.edu/%7Etcai/paper/Binomial-StatSci.pdf)\n13. [The difficult birth of stochastics: Jacob Bernoulli's Ars Conjectandi (1713)](https://www.sciencedirect.com/science/article/pii/S0315086014000482)\n14. [How well do we know Bernoulli? (smoothing for the multivariate Bernoulli NB classifier)](https://ceur-ws.org/Vol-835/paper5.pdf)\n15. [Multivariate Bernoulli distribution (Dai, Ding & Wahba)](https://pages.cs.wisc.edu/~sding/paper/mvb.pdf)\n16. [The Bernoulli model (Manning, Raghavan & Schütze, Introduction to Information Retrieval)](https://nlp.stanford.edu/IR-book/html/htmledition/the-bernoulli-model-1.html)\n17. [BernoulliNB, scikit-learn 1.9.0 documentation](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB)\n18. [Can binary logistic models be overdispersed? (Hilbe technical note)](http://www.highstat.com/Books/BGS/GLMGLMM/pdfs/HILBE-Can_binary_logistic_models_be_overdispersed2Jul2013.pdf)\n19. [The Validation of a Beta-Binomial Model for Overdispersed Binomial Data](https://pmc.ncbi.nlm.nih.gov/articles/PMC5736152/)\n20. [The continuous Bernoulli: fixing a pervasive error in variational autoencoders](https://proceedings.neurips.cc/paper/2019/file/f82798ec8909d23e55679ee26bb26437-Paper.pdf)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Statistical inference, estimation, sampling, and testing*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://pages.cs.wisc.edu/~sding/paper/mvb.pdf" ], "url": "https://www.edgepedia.org/bernoulli-model", "markdown_url": "https://www.edgepedia.org/bernoulli-model.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Bernoulli model\", Edgepedia (EdgeChat), https://www.edgepedia.org/bernoulli-model. Edgepedia Community License 1.0.", "credit_md": "\"[Bernoulli model](https://www.edgepedia.org/bernoulli-model)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/bernoulli-model](https://www.edgepedia.org/bernoulli-model). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Bernoulli model\", Edgepedia (EdgeChat), https://www.edgepedia.org/bernoulli-model. Edgepedia Community License 1.0.", "speakable": "The Bernoulli model is a statistical model for binary data where each observation is a success or failure with fixed success probability, underlying binomial and logistic models." }