PlanetMath (more info)
 Math for the people, by the people. Sponsor PlanetMath
Encyclopedia | Requests | Forums | Docs | Wiki | Random | RSS  
Login
create new user
name:
pass:
forget your password?
Main Menu
Owner confidence rating: Very high Entry average rating: No information on entry rating
differential entropy (Definition)

Let $(X, \mathfrak{B}, \mu)$ be a probability space, and let $f \in L^p(X, \mathfrak{B}, \mu)$ $||f||_{p} = 1$ be a function. The differential entropy $h(f)$ is defined as

\begin{equation} h(f) \equiv -\int_{X} |f|^p \log |f|^p\ d\mu \end{equation} Differential entropy is the continuous version of the Shannon entropy, $H[\mv{p}] = -\sum_{i} p_i \log p_i$ Consider first $u_a$ the uniform 1-dimensional distribution on $(0,a)$ The differential entropy is

\begin{equation} h(u_a) = -\int_{0}^{a} \frac{1}{a} \log \frac{1}{a}\ d\mu = \log a. \end{equation} Next consider probability distributions such as the function \begin{equation} g = \frac{1}{2 \pi \sigma}e^{-\frac{(t-\mu)^2}{2 \sigma^2}}, \end{equation}the 1-dimensional Gaussian. This pdf has differential entropy

\begin{equation} h(g) = -\int_{\mathbb{R}} g \log g\ dt = \frac{1}{2} \log 2 \pi e \sigma^2. \end{equation} For a general $n$ dimensional Gaussian $\mathcal{N}_{n}(\mv{\mu},\mv{K})$ with mean vector $\mv{\mu}$ and covariance matrix $\mv{K}$ $K_{ij} = \cov(x_i, x_j)$ we have

\begin{equation} h(\mathcal{N}_{n}(\mv{\mu},\mv{K})) = \frac{1}{2} \log (2 \pi e)^n |\mv{K}| \end{equation}where $|\mv{K}| = \det{\mv{K}}$




"differential entropy" is owned by Mathprof. [ full author list (3) | owner history (2) ]
(view preamble | get metadata)

View style:

See Also: Shannon's entropy, conditional entropy

Log in to rate this entry.
(view current ratings)

Cross-references: covariance matrix, mean vector, Gaussian, distribution, Shannon entropy, continuous, function, probability space
There are 2 references to this entry.

This is version 13 of differential entropy, born on 2002-02-13, modified 2006-12-09.
Object id is 1915, canonical name is DifferentialEntropy.
Accessed 9637 times total.

Classification:
AMS MSC54C70 (General topology :: Maps and general types of spaces defined by maps :: Entropy)

Pending Errata and Addenda
None.
[ View all 6 ]
Discussion
Style: Expand: Order:
forum policy

No messages.

Interact
post | correct | update request | add derivation | add example | add (any)