PlanetMath: relative entropy

(more info)

Math for the people, by the people.

Main Menu

sections Encyclopædia
Papers
Books
Expositions

meta Requests (200)
Orphanage
Unclass'd
Unproven (490)
Corrections (6)
Classification

talkback Polls
Forums
Feedback
Bug Reports

downloads Snapshots
PM Book

information News
Docs
Wiki
ChangeLog
TODO List
Legalese
About

relative entropy

(Definition)

Let and be probability distributions with supports $\mathcal{X}$ and $\mathcal{Y}$ respectively, where $\mathcal{X} \subset \mathcal{Y}$ . The relative entropy or Kullback-Leibler distance between two probability distributions and is defined as

$\displaystyle D(p\vert\vert q) :=\sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}.$

(1)

While $D(p\vert\vert q)$ is often called a distance, it is not a true metric because it is not symmetric and does not satisfy the triangle inequality. However, we do have $D(p\vert\vert q) \ge 0$ with equality iff .

$\displaystyle -D(p\vert\vert q)$	$\displaystyle = -\sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}$	(2)
	$\displaystyle = \sum_{x \in \mathcal{X}} p(x) \log \frac{q(x)}{p(x)}$	(3)
	$\displaystyle \le \log \left(\sum_{x \in \mathcal{X}} p(x) \frac{q(x)}{p(x)} \right)$	(4)
	$\displaystyle = \log \left(\sum_{x \in \mathcal{X}} q(x) \right)$	(5)
	$\displaystyle \le \log \left(\sum_{x \in \mathcal{Y}} q(x) \right)$	(6)
	$\displaystyle = 0$	(7)

where the first inequality follows from the concavity of $\log(x)$ and the second from expanding the sum over the support of

rather than

Relative entropy also comes in a continuous version which looks just as one might expect. For continuous distributions and , $\mathcal{S}$ the support of , we have

$\displaystyle D(f\vert\vert g) :=\int_{\mathcal{S}} f \log \frac{f}{g}.$

(8)

"relative entropy" is owned by Mathprof. [ full author list (3) | owner history (3) ]

(view preamble)

Other names:

Kullback-Leibler distance

Log in to rate this entry.
(view current ratings)

Cross-references: continuous, sum, inequality, iff, equality, triangle inequality, symmetric, metric, distance, supports, distributions
There are 2 references to this entry.

This is version 7 of relative entropy, born on 2002-02-13, modified 2006-09-21.
Object id is 1945, canonical name is RelativeEntropy.
Accessed 11684 times total.

Classification:

AMS MSC:	60E05 (Probability theory and stochastic processes :: Distribution theory :: Distributions: general theory)
	94A17 (Information and communication, circuits :: Communication, information :: Measures of information, entropy)

Pending Errata and Addenda

None.[ View all 7 ]

Discussion

forum policy

No messages.

Interact