|
|
|
|
relative entropy
|
(Definition)
|
|
|
Let $p$ and $q$ be probability distributions with supports $\mathcal{X}$ and $\mathcal{Y}$ respectively, where $ \mathcal{X} \subset \mathcal{Y}$ . The relative entropy or Kullback-Leibler distance between two probability distributions $p$ and $q$ is defined as
\begin{equation} D(p||q) \defined \sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}. \end{equation} While $D(p||q)$ is often called a distance, it is not a true metric because it is not symmetric and does not satisfy the triangle inequality. However, we do have $D(p||q) \ge 0$ with equality iff $p = q$ .
where the first inequality follows from the concavity of $\log(x)$ and the second from expanding the sum over the support of $q$ rather than $p$ .
Relative entropy also comes in a continuous version which looks just as one might expect. For continuous distributions $f$ and $g$ , $\mathcal{S}$ the support of $f$ , we have
\begin{equation} D(f||g) \defined \int_{\mathcal{S}} f \log \frac{f}{g}. \end{equation}
|
"relative entropy" is owned by Mathprof. [ full author list (3) | owner history (3) ]
|
|
(view preamble | get metadata)
Cross-references: continuous, sum, inequality, iff, equality, triangle inequality, symmetric, metric, distance, supports, distributions
There are 2 references to this entry.
This is version 7 of relative entropy, born on 2002-02-13, modified 2006-09-21.
Object id is 1945, canonical name is RelativeEntropy.
Accessed 13107 times total.
Classification:
| AMS MSC: | 60E05 (Probability theory and stochastic processes :: Distribution theory :: Distributions: general theory) | | | 94A17 (Information and communication, circuits :: Communication, information :: Measures of information, entropy) |
|
|
|
|
|
|
Pending Errata and Addenda
|
|
|
|
|
|
|
|
|
|
|