# Conditional entropy

**Conditional entropy** is a quantity in information theory that measures the average uncertainty remaining in a random variable Y when the value of another random variable X is known. It is written H(Y|X) and is expressed in bits when the logarithm base is 2. Claude E. Shannon first presented the concept in his 1948 work *A Mathematical Theory of Communication*, where he characterized it as a probability-weighted mean, over all values of x, of the entropy that y has when that particular x occurs<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup>. The paper appeared in the Bell System Technical Journal, Volume 27, pages 379–423 (July 1948) and pages 623–656 (October 1948)<sup>[2](https://spinchange.github.io/the-mathematical-theory-of-communication/)</sup>.

| Key fact | Detail |
|---|---|
| Definition | H(Y\|X) is the expected value of the entropy of Y computed under the conditional distributions of Y given each value of X<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup><sup> • </sup><sup>[3](https://opencw.aprende.org/courses/electrical-engineering-and-computer-science/6-441-information-theory-spring-2010/lecture-notes/MIT6_441S10_lec01.pdf)</sup> |
| Chain rule | H(X, Y) = H(X) + H(Y\|X): the uncertainty of the joint event is the uncertainty of X plus the uncertainty of Y when X is known<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup> |
| Conditioning reduces entropy | H(Y\|X) ≤ H(Y), with equality if and only if X and Y are independent<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup><sup> • </sup><sup>[4](https://stanforddatacompressionclass.github.io/Fall23/static_files/L8.pdf)</sup> |
| Non-negativity | For discrete variables the conditional entropy is non-negative, and it is zero when Y is a deterministic function of X<sup>[5](https://www.sciencedirect.com/topics/computer-science/thermodynamic-entropy)</sup><sup> • </sup><sup>[6](https://www.cl.cam.ac.uk/teaching/2021/InfoTheory/InfoTheoryNotes2020.pdf)</sup> |
| Role in channel capacity | Shannon called the conditional entropy H_y(x) the equivocation and defined the rate of transmission as R = H(x) − H_y(x)<sup>[7](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)</sup> |
| Fano's inequality | Relates the error probability P_e of guessing X from Y to H(X\|Y): P_e ≥ (H(X\|Y) − 1)/log \|A\| for an alphabet of \|A\| outcomes<sup>[6](https://www.cl.cam.ac.uk/teaching/2021/InfoTheory/InfoTheoryNotes2020.pdf)</sup> |
| Quantum case | The conditional von Neumann entropy can be negative for entangled quantum states; its negative is called the coherent information<sup>[8](https://www.cambridge.org/core/books/abs/quantum-information-theory/quantum-information-and-entropy/3D0DD775515E9C4A9B81A5602420F8AA)</sup> |

## Definition

For discrete random variables X and Y with outcomes in \( \mathcal{X} \) and \( \mathcal{Y} \), the conditional entropy of Y given X is the expectation, over the distribution of X, of the entropy of the conditional distribution of Y given X = x<sup>[9](https://statproofbook.github.io/D/ent-cond.html)</sup>. In summation form,

\[ H(Y|X) = -\sum_{x \in \mathcal{X}} \sum_{y \in \mathcal{Y}} p(x, y) \log p(y|x), \]

where p(x, y) is the joint probability mass function and p(y\|x) = p(x, y)/p(x) is the conditional probability<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup>. MIT lecture notes express the same idea as \( H(Y|X) = E_Z[H(Y|X=Z)] \), the average of the entropy of Y given X over all possible values of X<sup>[3](https://opencw.aprende.org/courses/electrical-engineering-and-computer-science/6-441-information-theory-spring-2010/lecture-notes/MIT6_441S10_lec01.pdf)</sup>.

Shannon's original notation was H_x(y), and he derived the identity

\[ H(x, y) = H(x) + H_x(y), \]

meaning that, given knowledge of x, the uncertainty attached to the combined event x, y equals that of x together with that of y<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup>. This chain rule extends to n variables as \( H(X_1, \ldots, X_n) = \sum_{i=1}^{n} H(X_i | X_1, \ldots, X_{i-1}) \)<sup>[3](https://opencw.aprende.org/courses/electrical-engineering-and-computer-science/6-441-information-theory-spring-2010/lecture-notes/MIT6_441S10_lec01.pdf)</sup>.

## Properties

**Conditioning reduces entropy.** From the chain rule and the non-negativity of discrete entropy, Shannon showed H(y) ≥ H_x(y): the uncertainty of y is never increased by knowledge of x, and it is decreased unless x and y are independent events, in which case it is not changed<sup>[1](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)</sup>. The Stanford data compression course states the same bound \( H(U|V) \le H(U) \) with equality if and only if U and V are independent<sup>[4](https://stanforddatacompressionclass.github.io/Fall23/static_files/L8.pdf)</sup>.

**Zero and non-negativity.** For discrete variables the conditional entropy is non-negative, and equality holds only when Y is a deterministic function of X (or X of Y)<sup>[5](https://www.sciencedirect.com/topics/computer-science/thermodynamic-entropy)</sup>. If X and Y are perfectly correlated, or X is any deterministic function of Y, then H(X\|Y) = 0, since no uncertainty about X remains once Y is known<sup>[6](https://www.cl.cam.ac.uk/teaching/2021/InfoTheory/InfoTheoryNotes2020.pdf)</sup>.

**Relation to mutual information.** [Mutual information](https://www.edgepedia.org/mutual-information), a measure of how dependent two random variables are, can be written in terms of conditional entropy as I(X; Y) = H(X) − H(X\|Y) = H(Y) − H(Y\|X)<sup>[3](https://opencw.aprende.org/courses/electrical-engineering-and-computer-science/6-441-information-theory-spring-2010/lecture-notes/MIT6_441S10_lec01.pdf)</sup>. It is the amount by which knowing one variable reduces the uncertainty of the other.

## Role in communication

Shannon used conditional entropy to define how much information a noisy channel actually transmits. In his example of a binary source producing 1000 bits per second with about 1 in 100 symbols received incorrectly, he argued that the proper correction is the uncertainty about what was sent given what was received. He defined the rate of transmission as

\[ R = H(x) - H_y(x), \]

where the conditional entropy H_y(x), which he named the equivocation, measures the average ambiguity of the received signal<sup>[7](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)</sup>. For a noisy channel, the joint entropy of input and output satisfies H(x, y) = H(x) + H_x(y) = H(y) + H_y(x), and the two conditional entropies H_x(y) and H_y(x) are the entropy of the output when the input is known and conversely<sup>[7](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)</sup>.

Conditional entropy also enters bounds on guessing. [Fano's inequality](https://www.edgepedia.org/fanos-inequality) connects the error probability P_e incurred in deducing X from Y with the conditional entropy H(X\|Y), assuming \|A\| possible outcomes.<sup>[6](https://www.cl.cam.ac.uk/teaching/2021/InfoTheory/InfoTheoryNotes2020.pdf)</sup>

\[ P_e \ge \frac{H(X|Y) - 1}{\log |\mathcal{A}|}. \]

In statistical estimation, Fano's inequality bounds the error of any estimator through the conditional entropy H(V\|V̂) along a Markov chain V → X → V̂<sup>[10](https://web.stanford.edu/class/stats311/lecture-notes.pdf)</sup>.

## Extensions

**Continuous variables.** For random variables with probability density functions, the conditional differential entropy is defined as h(X\|Y) = −E[log f(X\|Y)], and mutual information becomes I(X; Y) = h(X) − h(X\|Y)<sup>[11](https://www.sciencedirect.com/topics/computer-science/differential-entropy)</sup>. The chain rules still hold for differential entropy, and conditioning still reduces entropy because differential entropy is concave in the input distribution<sup>[12](https://ocw.mit.edu/courses/6-441-information-theory-spring-2010/c98ee16aa0e4e01fb6719e7064af8042_MIT6_441S10_lec16.pdf)</sup>. Unlike discrete entropy, differential entropy can be positive or negative<sup>[13](https://web.stanford.edu/class/ee376a/files/scribes/lecture10.pdf)</sup>.

**Quantum systems.** The quantum analog uses the von Neumann entropy. A radical departure from the classical case appears there: the conditional quantum entropy can be negative for certain quantum states, and the negative of the conditional quantum entropy is sufficiently important in quantum information theory that it has a special name, the coherent information<sup>[8](https://www.cambridge.org/core/books/abs/quantum-information-theory/quantum-information-and-entropy/3D0DD775515E9C4A9B81A5602420F8AA)</sup>. Conditional entropies can be negative for entangled systems such as an Einstein-Podolsky-Rosen pair, which violates bounds of classical information theory<sup>[14](https://arxiv.org/abs/quant-ph/9610005)</sup>. In quantum systems, entropy and information depend on the choice of measurements performed over the systems, a fundamental difference from the classical case<sup>[15](https://www.sciencedirect.com/science/article/pii/S0960077998002252)</sup>.

**Estimation from data.** When the joint distribution is unknown, conditional entropy must be estimated from samples. Bulinski and Kozhevin introduced estimates of the conditional Shannon entropy for a discrete response variable depending on a vector of d factors, constructed by means of spatial order statistics, or k-nearest-neighbor statistics where k = k_n depends on the number of observations n, and established asymptotic unbiasedness and L²-consistency under simple conditions<sup>[16](https://arxiv.org/abs/1804.08741)</sup>. For finitely supported discrete variables, plug-in estimators of joint entropy, conditional entropy, and mutual information carry a finite-sample bias that has been studied with bias-corrected constructions<sup>[17](https://arxiv.org/abs/2610.02558)</sup>.

## Interpretation

The conditional entropy answers a practical question: how many bits per symbol are still needed to describe Y once X is known. In source modeling, the conditional entropy of the next symbol when the preceding N − 1 symbols are known is the entropy of the Nth-order approximation to a source, and it equals the source entropy when statistical influences extend no more than N symbols<sup>[7](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)</sup>. In dynamical systems, the conditional entropy of one finite partition of a probability space given another is interpreted as the amount of information added by introducing the first partition when the second is already known<sup>[18](https://www.scholarpedia.org/article/Entropy)</sup>.

## References

1. [Shannon, C. E. (1948). A Mathematical Theory of Communication, Bell System Technical Journal (part I transcription)](https://academicweb.nd.edu/~powers/ame.20231/shannon1948a.pdf)
2. [A Mathematical Theory of Communication — Claude E. Shannon (1948), publication details](https://spinchange.github.io/the-mathematical-theory-of-communication/)
3. [MIT 6.441 Information Theory, Lecture 1 notes](https://opencw.aprende.org/courses/electrical-engineering-and-computer-science/6-441-information-theory-spring-2010/lecture-notes/MIT6_441S10_lec01.pdf)
4. [Properties of conditional entropy, Stanford Data Compression course, Lecture 8](https://stanforddatacompressionclass.github.io/Fall23/static_files/L8.pdf)
5. [Thermodynamic Entropy — ScienceDirect Topics](https://www.sciencedirect.com/topics/computer-science/thermodynamic-entropy)
6. [Information Theory lecture notes, University of Cambridge (2021)](https://www.cl.cam.ac.uk/teaching/2021/InfoTheory/InfoTheoryNotes2020.pdf)
7. [Shannon, C. E. (1948). A Mathematical Theory of Communication, full text](https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)
8. [Quantum Information and Entropy, in Quantum Information Theory, Cambridge University Press](https://www.cambridge.org/core/books/abs/quantum-information-theory/quantum-information-and-entropy/3D0DD775515E9C4A9B81A5602420F8AA)
9. [Definition: Conditional entropy, The Book of Statistical Proofs](https://statproofbook.github.io/D/ent-cond.html)
10. [Duchi, J. Lecture Notes on Statistics and Information Theory, Stanford Statistics 311](https://web.stanford.edu/class/stats311/lecture-notes.pdf)
11. [Differential Entropy — ScienceDirect Topics](https://www.sciencedirect.com/topics/computer-science/differential-entropy)
12. [MIT 6.441 Information Theory, Lecture 16: Differential entropy](https://ocw.mit.edu/courses/6-441-information-theory-spring-2010/c98ee16aa0e4e01fb6719e7064af8042_MIT6_441S10_lec16.pdf)
13. [EE376A Information Theory, Stanford, Lecture 10: Conditional differential entropy](https://web.stanford.edu/class/ee376a/files/scribes/lecture10.pdf)
14. [Negative entropy in quantum information theory, arXiv:quant-ph/9610005](https://arxiv.org/abs/quant-ph/9610005)
15. [Conditional Entropy and Information in Quantum Systems, ScienceDirect](https://www.sciencedirect.com/science/article/pii/S0960077998002252)
16. [Bulinski, A., Kozhevin, A. (2018). Statistical Estimation of Conditional Shannon Entropy, arXiv:1804.08741](https://arxiv.org/abs/1804.08741)
17. [Bias-Corrected Estimators for Joint Entropy, Conditional Entropy, and Mutual Information in the Discrete Bivariate Case, arXiv:2610.02558](https://arxiv.org/abs/2610.02558)
18. [Entropy, Scholarpedia](https://www.scholarpedia.org/article/Entropy)

---
*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Probability theory › Conditional probability and independence › Conditional probability*

*Initially written Sep 17, 2026 · Reviewed: Oct 15, 2026 · Edited: Oct 15, 2026 · Last review: Oct 15, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license