## You are here

Homedifferential entropy

## Primary tabs

# differential entropy

Let $(X,\mathfrak{B},\mu)$ be a probability space, and let $f\in L^{p}(X,\mathfrak{B},\mu)$, $||f||_{{p}}=1$ be a function. The *differential entropy* $h(f)$ is defined as

$h(f)\equiv-\int_{{X}}|f|^{p}\log|f|^{p}\ d\mu$ | (1) |

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}]=-\sum_{{i}}p_{i}\log p_{i}$. Consider first $u_{a}$, the uniform 1-dimensional distribution on $(0,a)$. The differential entropy is

$h(u_{a})=-\int_{{0}}^{{a}}\frac{1}{a}\log\frac{1}{a}\ d\mu=\log a.$ | (2) |

Next consider probability distributions such as the function

$g=\frac{1}{2\pi\sigma}e^{{-\frac{(t-\mu)^{2}}{2\sigma^{2}}}},$ | (3) |

the 1-dimensional Gaussian. This pdf has differential entropy

$h(g)=-\int_{{\mathbb{R}}}g\log g\ dt=\frac{1}{2}\log 2\pi e\sigma^{2}.$ | (4) |

For a general $n$-dimensional Gaussian $\mathcal{N}_{{n}}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$, $K_{{ij}}=\mathrm{cov}(x_{i},x_{j})$, we have

$h(\mathcal{N}_{{n}}(\mathbf{\mu},\mathbf{K}))=\frac{1}{2}\log(2\pi e)^{n}|% \mathbf{K}|$ | (5) |

where $|\mathbf{K}|=\det{\mathbf{K}}$.

## Mathematics Subject Classification

54C70*no label found*

- Forums
- Planetary Bugs
- HS/Secondary
- University/Tertiary
- Graduate/Advanced
- Industry/Practice
- Research Topics
- LaTeX help
- Math Comptetitions
- Math History
- Math Humor
- PlanetMath Comments
- PlanetMath System Updates and News
- PlanetMath help
- PlanetMath.ORG
- Strategic Communications Development
- The Math Pub
- Testing messages (ignore)

- Other useful stuff

## Recent Activity

new correction: Error in proof of Proposition 2 by alex2907

Jun 24

new question: A good question by Ron Castillo

Jun 23

new question: A trascendental number. by Ron Castillo

Jun 19

new question: Banach lattice valued Bochner integrals by math ias

Jun 13

new question: young tableau and young projectors by zmth

Jun 11

new question: binomial coefficients: is this a known relation? by pfb