proof of Gaussian maximizes entropy for given covariance

Let $f,K,\varphi$ be as in the parent (http://planetmath.org/GaussianMaximizesEntropyForGivenCovariance) entry.

The proof uses the nonnegativity of relative entropy $D(f||\varphi )$, and an interesting property of quadratic forms. If $A$ is a quadratic form and $p,q$ are probability distributions each with mean $\mathrm{𝟎}$ and covariance matrix $𝐊$, we have

$$\int p{x}_i{x}_j𝑑x_i𝑑x_j=K_{ij}=\int q{x}_i{x}_j𝑑x_i𝑑x_j$$ (1)

and thus

$$\int Ap=\int Aq$$ (2)

Now note that since

$$\varphi (𝐱)={\left({(2\pi )}^n|𝐊|\right)}^{-\frac{1}{2}}\exp (-\frac{1}{2}{𝐱}^{\mathrm{T}}{𝐊}^{-1}𝐱),$$ (3)

we see that $\log \varphi$ is a quadratic form plus a constant.

0 $\le D(f||\varphi )$

$={\displaystyle \int f\log \frac{f}{\varphi }}$

$={\displaystyle \int f\log f}-{\displaystyle \int f\log \varphi }$

$=-h(f)-{\displaystyle \int f\log \varphi }$

$=-h(f)-{\displaystyle \int \varphi \log \varphi }\mathit{\hspace{1em}\hspace{1em}}\text{by the quadratic form property above}$

$=-h(f)+h(\varphi )$

and thus $h(\varphi )\ge h(f)$.

Title	proof of Gaussian maximizes entropy for given covariance
Canonical name	ProofOfGaussianMaximizesEntropyForGivenCovariance
Date of creation	2013-03-22 12:19:35
Last modified on	2013-03-22 12:19:35
Owner	Mathprof (13753)
Last modified by	Mathprof (13753)
Numerical id	10
Author	Mathprof (13753)
Entry type	Proof
Classification	msc 94A17
Related topic	QuadraticForm
Related topic	RelativeEntropy
Related topic	MultidimensionalGaussianIntegral