|
|
|
|
mutual information
|
(Definition)
|
|
|
Let $(\Omega, \mathcal{F}, \mu)$ be a discrete probability space, and let $X$ and $Y$ be discrete random variables on $\Omega$ .
The mutual information $I[X;Y]$ , read as ``the mutual information of $X$ and $Y$ ,'' is defined as
where $D$ denotes the relative entropy.
Mutual information, or just information, is measured in bits if the logarithm is to the base 2, and in ``nats'' when using the natural logarithm.
The most obvious characteristic of mutual information is that it depends on both $X$ and $Y$ . There is no information in a vacuum--information is always about something. In this case, $I[X;Y]$ is the information in $X$ about $Y$ . As its name suggests, mutual information is symmetric, $I[X;Y] = I[Y;X]$ , so any information $X$ carries about $Y$ , $Y$ also carries about $X$ .
The definition in terms of relative entropy gives a useful interpretation of $I[X;Y]$ as a kind of ``distance'' between the joint distribution $\mu(x,y)$ and the product distribution $\mu(x)\mu(y)$ . Recall, however, that relative entropy is not a true distance, so this is just a conceptual tool. However,
it does capture another intuitive notion of information. Remember that for $X,Y$ independent, $\mu(x,y) = \mu(x)\mu(y)$ . Thus the relative entropy ``distance'' goes to zero, and we have $I[X;Y]=0$ as one would expect for independent random variables.
A number of useful expressions, most apparent from the definition, relate mutual information to the entropy $H$ :
Recall that the entropy $H[X]$ quantifies our uncertainty about $X$ . The last line justifies the description of entropy as ``self-information.''
Mutual information, or simply information, was introduced by Shannon in his landmark 1948 paper ``A Mathematical Theory of Communication.''
|
"mutual information" is owned by drummond.
|
|
(view preamble | get metadata)
Cross-references: line, entropy, expressions, number, random variables, independent, distance, distribution, product, joint distribution, interpretation, terms, symmetric, characteristic, obvious, natural logarithm, base, logarithm, relative entropy, discrete random variables, probability space, discrete
There are 93 references to this entry.
This is version 1 of mutual information, born on 2002-04-30.
Object id is 2887, canonical name is MutualInformation.
Accessed 29600 times total.
Classification:
| AMS MSC: | 94A17 (Information and communication, circuits :: Communication, information :: Measures of information, entropy) |
|
|
|
|
|
|
Pending Errata and Addenda
|
|
|
|
|
|
|
|
|
|
|
![$\displaystyle I[X;Y]$](http://images.planetmath.org:8080/cache/objects/2887/js/img1.png "$\displaystyle I[X;Y]$"){kind=small}
![$\displaystyle 0 \le I[X;Y]$](http://images.planetmath.org:8080/cache/objects/2887/js/img4.png "$\displaystyle 0 \le I[X;Y]$"){kind=small}
![$\displaystyle \le H[X]$](http://images.planetmath.org:8080/cache/objects/2887/js/img5.png "$\displaystyle \le H[X]$"){kind=small}
![$\displaystyle I[X;Y]$](http://images.planetmath.org:8080/cache/objects/2887/js/img6.png "$\displaystyle I[X;Y]$"){kind=small}
![$\displaystyle = H[X] - H[X\vert Y]$](http://images.planetmath.org:8080/cache/objects/2887/js/img7.png "$\displaystyle = H[X] - H[X\vert Y]$"){kind=small}
![$\displaystyle I[X;Y]$](http://images.planetmath.org:8080/cache/objects/2887/js/img8.png "$\displaystyle I[X;Y]$"){kind=small}
![$\displaystyle = H[X] + H[Y] - H[X,Y]$](http://images.planetmath.org:8080/cache/objects/2887/js/img9.png "$\displaystyle = H[X] + H[Y] - H[X,Y]$"){kind=small}
![$\displaystyle I[X;X]$](http://images.planetmath.org:8080/cache/objects/2887/js/img10.png "$\displaystyle I[X;X]$"){kind=small}
![$\displaystyle = H[X]$](http://images.planetmath.org:8080/cache/objects/2887/js/img11.png "$\displaystyle = H[X]$"){kind=small}