PlanetMath: differential entropy

(more info)

Math for the people, by the people.

Main Menu

sections Encyclopædia
Papers
Books
Expositions

meta Requests (210)
Orphanage
Unclass'd (1)
Unproven (472)
Corrections (37)
Classification

talkback Polls
Forums
Feedback
Bug Reports

downloads Snapshots
PM Book

information News
Docs
Wiki
ChangeLog
TODO List
Legalese
About

Entry average rating: No information on entry rating

differential entropy

(Definition)

Let $(X, \mathfrak{B}, \mu)$ be a probability space, and let $f \in L^p(X, \mathfrak{B}, \mu)$ , $\vert\vert f\vert\vert _{p} = 1$ be a function. The differential entropy is defined as

$\displaystyle h(f) \equiv -\int_{X} \vert f\vert^p \log \vert f\vert^p\ d\mu$

(1)

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}] = -\sum_{i} p_i \log p_i$ . Consider first , the uniform 1-dimensional distribution on . The differential entropy is

$\displaystyle h(u_a) = -\int_{0}^{a} \frac{1}{a} \log \frac{1}{a}\ d\mu = \log a.$

(2)

Next consider probability distributions such as the function

$\displaystyle g = \frac{1}{2 \pi \sigma}e^{-\frac{(t-\mu)^2}{2 \sigma^2}},$

(3)

the 1-dimensional Gaussian. This pdf has differential entropy

$\displaystyle h(g) = -\int_{\mathbb{R}} g \log g\ dt = \frac{1}{2} \log 2 \pi e \sigma^2.$

(4)

For a general -dimensional Gaussian $\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$ , $K_{ij} = \mathrm{cov}(x_i, x_j)$ , we have

$\displaystyle h(\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K})) = \frac{1}{2} \log (2 \pi e)^n \vert\mathbf{K}\vert$

(5)

where $\vert\mathbf{K}\vert = \det{\mathbf{K}}$ .

"differential entropy" is owned by Mathprof. [ full author list (3) | owner history (2) ]

(view preamble)

Log in to rate this entry.
(view current ratings)

Cross-references: covariance matrix, mean vector, Gaussian, distribution, Shannon entropy, continuous, function, probability space
There are 2 references to this entry.

This is version 13 of differential entropy, born on 2002-02-13, modified 2006-12-09.
Object id is 1915, canonical name is DifferentialEntropy.
Accessed 7679 times total.

Classification:

AMS MSC:

54C70 (General topology :: Maps and general types of spaces defined by maps :: Entropy)

Pending Errata and Addenda

None.[ View all 6 ]

Discussion

forum policy

No messages.

Interact