## You are here

Homedifferential entropy

## Primary tabs

# differential entropy

Let $(X,\mathfrak{B},\mu)$ be a probability space, and let $f\in L^{p}(X,\mathfrak{B},\mu)$, $||f||_{{p}}=1$ be a function. The *differential entropy* $h(f)$ is defined as

$h(f)\equiv-\int_{{X}}|f|^{p}\log|f|^{p}\ d\mu$ | (1) |

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}]=-\sum_{{i}}p_{i}\log p_{i}$. Consider first $u_{a}$, the uniform 1-dimensional distribution on $(0,a)$. The differential entropy is

$h(u_{a})=-\int_{{0}}^{{a}}\frac{1}{a}\log\frac{1}{a}\ d\mu=\log a.$ | (2) |

Next consider probability distributions such as the function

$g=\frac{1}{2\pi\sigma}e^{{-\frac{(t-\mu)^{2}}{2\sigma^{2}}}},$ | (3) |

the 1-dimensional Gaussian. This pdf has differential entropy

$h(g)=-\int_{{\mathbb{R}}}g\log g\ dt=\frac{1}{2}\log 2\pi e\sigma^{2}.$ | (4) |

For a general $n$-dimensional Gaussian $\mathcal{N}_{{n}}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$, $K_{{ij}}=\mathrm{cov}(x_{i},x_{j})$, we have

$h(\mathcal{N}_{{n}}(\mathbf{\mu},\mathbf{K}))=\frac{1}{2}\log(2\pi e)^{n}|% \mathbf{K}|$ | (5) |

where $|\mathbf{K}|=\det{\mathbf{K}}$.

## Mathematics Subject Classification

54C70*no label found*

- Forums
- Planetary Bugs
- HS/Secondary
- University/Tertiary
- Graduate/Advanced
- Industry/Practice
- Research Topics
- LaTeX help
- Math Comptetitions
- Math History
- Math Humor
- PlanetMath Comments
- PlanetMath System Updates and News
- PlanetMath help
- PlanetMath.ORG
- Strategic Communications Development
- The Math Pub
- Testing messages (ignore)

- Other useful stuff

## Recent Activity

new question: Prove that for any sets A, B, and C, An(BUC)=(AnB)U(AnC) by St_Louis

Apr 20

new image: information-theoretic-distributed-measurement-dds.png by rspuzio

new image: information-theoretic-distributed-measurement-4.2 by rspuzio

new image: information-theoretic-distributed-measurement-4.1 by rspuzio

new image: information-theoretic-distributed-measurement-3.2 by rspuzio

new image: information-theoretic-distributed-measurement-3.1 by rspuzio

new image: information-theoretic-distributed-measurement-2.1 by rspuzio

Apr 19

new collection: On the Information-Theoretic Structure of Distributed Measurements by rspuzio

Apr 15

new question: Prove a formula is part of the Gentzen System by LadyAnne

Mar 30

new question: A problem about Euler's totient function by mbhatia