differential entropy

Let $(X,\mathfrak{B},\mu)$ be a probability space, and let $f\in L^{p}(X,\mathfrak{B},\mu)$ , $||f||_{p}=1$ be a function. The differential entropy $h(f)$ is defined as

h(f)\equiv-\int_{X}|f|^{p}\log|f|^{p}\ d\mu

(1)

Differential entropy is the continuous version of the Shannon entropy, $H[\mathbf{p}]=-\sum_{i}p_{i}\log p_{i}$ . Consider first $u_{a}$ , the uniform 1-dimensional distribution on $(0,a)$ . The differential entropy is

h(u_{a})=-\int_{0}^{a}\frac{1}{a}\log\frac{1}{a}\ d\mu=\log a.

(2)

Next consider probability distributions such as the function

g=\frac{1}{2\pi\sigma}e^{-\frac{(t-\mu)^{2}}{2\sigma^{2}}},

(3)

the 1-dimensional Gaussian. This pdf has differential entropy

h(g)=-\int_{\mathbb{R}}g\log g\ dt=\frac{1}{2}\log 2\pi e\sigma^{2}.

(4)

For a general $n$ -dimensional Gaussian (http://planetmath.org/JointNormalDistribution) $\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K})$ with mean vector $\mathbf{\mu}$ and covariance matrix $\mathbf{K}$ , $K_{ij}=\mathrm{cov}(x_{i},x_{j})$ , we have

h(\mathcal{N}_{n}(\mathbf{\mu},\mathbf{K}))=\frac{1}{2}\log(2\pi e)^{n}|% \mathbf{K}|

(5)

where $|\mathbf{K}|=\det{\mathbf{K}}$ .

Title	differential entropy
Canonical name	DifferentialEntropy
Date of creation	2013-03-22 12:18:48
Last modified on	2013-03-22 12:18:48
Owner	Mathprof (13753)
Last modified by	Mathprof (13753)
Numerical id	16
Author	Mathprof (13753)
Entry type	Definition
Classification	msc 54C70
Related topic	ShannonsTheoremEntropy
Related topic	ConditionalEntropy