|
Let $(X, \mathfrak{B}, \mu)$ be a probability space, and let $f \in L^p(X, \mathfrak{B}, \mu)$ $||f||_{p} = 1$ be a function. The differential entropy $h(f)$ is defined as
\begin{equation} h(f) \equiv -\int_{X} |f|^p \log |f|^p\ d\mu \end{equation} Differential entropy is the continuous version of the Shannon entropy, $H[\mv{p}] = -\sum_{i} p_i \log p_i$ Consider first $u_a$ the uniform 1-dimensional distribution on $(0,a)$ The differential entropy is
\begin{equation} h(u_a) = -\int_{0}^{a} \frac{1}{a} \log \frac{1}{a}\ d\mu = \log a. \end{equation} Next consider probability distributions such as the function \begin{equation} g = \frac{1}{2 \pi \sigma}e^{-\frac{(t-\mu)^2}{2 \sigma^2}}, \end{equation}the 1-dimensional Gaussian. This pdf has differential entropy
\begin{equation} h(g) = -\int_{\mathbb{R}} g \log g\ dt = \frac{1}{2} \log 2 \pi e \sigma^2. \end{equation} For a general $n$ dimensional Gaussian $\mathcal{N}_{n}(\mv{\mu},\mv{K})$ with mean vector $\mv{\mu}$ and covariance matrix $\mv{K}$ $K_{ij} = \cov(x_i, x_j)$ we have
\begin{equation} h(\mathcal{N}_{n}(\mv{\mu},\mv{K})) = \frac{1}{2} \log (2 \pi e)^n |\mv{K}| \end{equation}where $|\mv{K}| = \det{\mv{K}}$
|