proof of Gaussian maximizes entropy for given covariance


Let f,K,ϕ be as in the parent (http://planetmath.org/GaussianMaximizesEntropyForGivenCovariance) entry.

The proof uses the nonnegativity of relative entropyMathworldPlanetmath D(f||ϕ), and an interesting property of quadratic formsMathworldPlanetmath. If A is a quadratic form and p,q are probability distributions each with mean 𝟎 and covariance matrix 𝐊, we have

pxixj𝑑xi𝑑xj=Kij=qxixj𝑑xi𝑑xj (1)

and thus

Ap=Aq (2)

Now note that since

ϕ(𝐱)=((2π)n|𝐊|)-12exp(-12𝐱T𝐊-1𝐱), (3)

we see that logϕ is a quadratic form plus a constant.

0 D(f||ϕ)
=flogfϕ
=flogf-flogϕ
=-h(f)-flogϕ
=-h(f)-ϕlogϕ  by the quadratic form property above
=-h(f)+h(ϕ)

and thus h(ϕ)h(f).

Title proof of Gaussian maximizes entropy for given covariance
Canonical name ProofOfGaussianMaximizesEntropyForGivenCovariance
Date of creation 2013-03-22 12:19:35
Last modified on 2013-03-22 12:19:35
Owner Mathprof (13753)
Last modified by Mathprof (13753)
Numerical id 10
Author Mathprof (13753)
Entry type Proof
Classification msc 94A17
Related topic QuadraticForm
Related topic RelativeEntropy
Related topic MultidimensionalGaussianIntegral