PlanetMath: conditional entropy

(more info)

Math for the people, by the people.

donor list - find out how

Main Menu

sections Encyclopædia
Papers
Books
Expositions

meta Requests (236)
Orphanage
Unclass'd (1)
Unproven (540)
Corrections (51)
Classification

talkback Polls
Forums
Feedback
Bug Reports

downloads Snapshots
PM Book

information News
Docs
Wiki
ChangeLog
TODO List
Copyright
About

Entry average rating: No information on entry rating

conditional entropy

(Definition)

Definition (Discrete)

Let $(\Omega, \mathcal{F}, \mu)$ be a discrete probability space, and let $X$ and $Y$ be discrete random variables on $\Omega$ .

The conditional entropy $H[X|Y]$ , read as ``the conditional entropy of $X$ given $Y$ ,'' is defined as \begin{equation} H[X|Y] = -\sum_{x \in X}\sum_{y \in Y} \mu(X=x,Y=y) \log \mu(X=x|Y=y) \end{equation}where $\mu(X|Y)$ denotes the conditional probability. $\mu(Y=y)$ is nonzero in the discrete case

Discussion

The results for discrete conditional entropy will be assumed to hold for the continuous case unless we indicate otherwise.

With $H[X,Y]$ the joint entropy and $f$ a function, we have the following results:

$\displaystyle H[X\vert Y] + H[Y]$	$\displaystyle = H[X,Y]$	(1)
$\displaystyle H[X\vert Y]$	$\displaystyle \le H[X]$ (conditioning reduces entropy)	(2)
$\displaystyle H[X\vert Y]$	$\displaystyle \le H[X] + H[Y]$ (equality iff $\displaystyle X, Y$ independent)	(3)
$\displaystyle H[X\vert Y]$	$\displaystyle \le H[X\vert f(Y)]$	(4)
$\displaystyle H[X\vert Y]$	$\displaystyle = 0 \iff X=f(Y)$ (special case $\displaystyle H[X\vert X] = 0$ )	(5)

The conditional entropy $H[X|Y]$ may be interpreted as the uncertainty in $X$ given knowledge of $Y$ . (Try reading the above equalities and inequalities with this interpretation in mind.)

"conditional entropy" is owned by PrimeFan. [ full author list (3) | owner history (6) ]

(view preamble | get metadata)

Log in to rate this entry.
(view current ratings)

Cross-references: interpretation, inequalities, equalities, function, joint entropy, continuous, conditional probability, discrete random variables, probability space, discrete

This is version 6 of conditional entropy, born on 2002-02-21, modified 2006-10-30.
Object id is 2363, canonical name is ConditionalEntropy.
Accessed 11890 times total.

Classification:

AMS MSC:

94A17 (Information and communication, circuits :: Communication, information :: Measures of information, entropy)

Pending Errata and Addenda

None.[ View all 4 ]

Discussion

forum policy

No messages.

Interact