## You are here

Homedifferential entropy

## Primary tabs

# differential entropy

Let $(X,\mathfrak{B},\mu)$ be a probability space, and let $f\in L^{p}(X,\mathfrak{B},\mu)$, $||f||_{{p}}=1$ be a function. The *differential entropy* $h(f)$ is defined as

$h(f)\equiv-\int_{{X}}|f|^{p}\log|f|^{p}\ d\mu$ | (1) |

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}]=-\sum_{{i}}p_{i}\log p_{i}$. Consider first $u_{a}$, the uniform 1-dimensional distribution on $(0,a)$. The differential entropy is

$h(u_{a})=-\int_{{0}}^{{a}}\frac{1}{a}\log\frac{1}{a}\ d\mu=\log a.$ | (2) |

Next consider probability distributions such as the function

$g=\frac{1}{2\pi\sigma}e^{{-\frac{(t-\mu)^{2}}{2\sigma^{2}}}},$ | (3) |

the 1-dimensional Gaussian. This pdf has differential entropy

$h(g)=-\int_{{\mathbb{R}}}g\log g\ dt=\frac{1}{2}\log 2\pi e\sigma^{2}.$ | (4) |

For a general $n$-dimensional Gaussian $\mathcal{N}_{{n}}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$, $K_{{ij}}=\mathrm{cov}(x_{i},x_{j})$, we have

$h(\mathcal{N}_{{n}}(\mathbf{\mu},\mathbf{K}))=\frac{1}{2}\log(2\pi e)^{n}|% \mathbf{K}|$ | (5) |

where $|\mathbf{K}|=\det{\mathbf{K}}$.

## Mathematics Subject Classification

54C70*no label found*

- Forums
- Planetary Bugs
- HS/Secondary
- University/Tertiary
- Graduate/Advanced
- Industry/Practice
- Research Topics
- LaTeX help
- Math Comptetitions
- Math History
- Math Humor
- PlanetMath Comments
- PlanetMath System Updates and News
- PlanetMath help
- PlanetMath.ORG
- Strategic Communications Development
- The Math Pub
- Testing messages (ignore)

- Other useful stuff

## Recent Activity

new question: Prime numbers out of sequence by Rubens373

Oct 7

new question: Lorenz system by David Bankom

Oct 19

new correction: examples and OEIS sequences by fizzie

Oct 13

new correction: Define Galois correspondence by porton

Oct 7

new correction: Closure properties on languages: DCFL not closed under reversal by babou

new correction: DCFLs are not closed under reversal by petey

Oct 2

new correction: Many corrections by Smarandache

Sep 28

new question: how to contest an entry? by zorba

new question: simple question by parag