conditional distribution of multi-variate normal variable

Theorem.

Let $X$ be a random variable, taking values in ${\mathrm{R}}^n$, normally distributed with a non-singular covariance matrix $\mathrm{\Sigma }$ and a mean of zero.

Suppose $Y$ is defined by $Y\mathrm{=}{B}^{\mathrm{*}}\mathit{}X$ for some linear transformation $B\mathrm{:}{\mathrm{R}}^k\mathrm{\to }{\mathrm{R}}^n$ of maximum rank. (${}^{\mathrm{\ast }}$ to denotes the transpose operator.)

Then the distribution of $X$ conditioned on $Y$ is multi-variate normal, with conditional means and covariances of:

$$𝔼[X\mid Y]=\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}Y,\mathrm{Var}[X\mid Y]=\mathrm{\Sigma }-\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}{(\mathrm{\Sigma }B)}^{\ast }.$$

If $k\mathrm{=}\mathrm{1}$, so that $B$ is simply a vector in ${\mathrm{R}}^n$, these formulas reduce to:

$$𝔼[X\mid Y]=\frac{\mathrm{\Sigma }BY}{\mathrm{Var}[Y]},\mathrm{Var}[X\mid Y]=\mathrm{\Sigma }-\frac{\mathrm{\Sigma }B{B}^{\ast }\mathrm{\Sigma }}{\mathrm{Var}[Y]}.$$

If $X$ does not have zero mean, then the formula for $\mathrm{E}\mathrm{[}X\mathrm{\mid }Y\mathrm{]}$ is modified by adding $\mathrm{E}\mathit{}\mathrm{[}X\mathrm{]}$ and replacing $Y$ by $Y\mathrm{-}\mathrm{E}\mathit{}\mathrm{[}Y\mathrm{]}$, and the formula for $\mathrm{Var}\mathit{}\mathrm{[}X\mathrm{\mid }Y\mathrm{]}$ is unchanged.

Proof.

We split up $X$ into two stochastically independent parts, the first part containing exactly the information embodied in $Y$. Then the conditional distribution of $X$ given $Y$ is simply the unconditional distribution of the second part that is independent of $Y$.

To this end, we first change variables to express everything in terms of a standard multi-variate normal $Z$. Let $A:{ℝ}^n\to {ℝ}^n$ be a “square root” factorization of the covariance matrix $\mathrm{\Sigma }$, so that:

$$A{A}^{\ast }=\mathrm{\Sigma },Z=A^{-1}X,X=AZ,Y=B^{\ast }AZ.$$

We let $H:{ℝ}^n\to {ℝ}^n$ be the orthogonal projection onto the range of $A^{\ast }B:{ℝ}^k\to {ℝ}^n$, and decompose $Z$ into orthogonal components:

$$Z=HZ+(I-H)Z.$$

It is intuitively obvious that orthogonality of the two random normal vectors implies their stochastic independence. To show this formally, observe that the Gaussian density function for $Z$ factors into a product:

$${(2\pi )}^{-n/2}\exp \left(-\frac{1}{2}{\parallel z\parallel }^2\right)={(2\pi )}^{-n/2}\exp \left(-\frac{1}{2}{\parallel Hz\parallel }^2\right)\exp \left(-\frac{1}{2}{\parallel (I-H)z\parallel }^2\right).$$

We can construct an orthonormal system of coordinates on ${ℝ}^n$ under which the components for $Hz$ are completely disjoint from those components of $(I-H)z$. On the other hand, the densities for $Z$, $HZ$, and $(I-H)Z$ remain invariant even after changing coordinates, because they are radially symmetric. Hence the variables $HZ$ and $(I-H)Z$ are separable in their joint density and they are independent.

$HZ$ embodies the information in the linear combination $Y=B^{\ast }AZ$. For we have the identity:

$$Y=\left(B^{\ast }A\right)Z=\left(B^{\ast }A\right)\left(HZ+(I-H)Z\right)=\left(B^{\ast }A\right)HZ+0.$$

The last term is null because $(I-H)Z$ is orthogonal to the range of $A^{\ast }B$ by definition. (Equivalently, $(I-H)Z$ lies in the kernel of ${(A^{\ast }B)}^{\ast }=B^{\ast }A$.) Thus $Y$ can always be recovered by a linear transformation on $HZ$.

Conversely, $Y$ completely determines $HZ$, from the analytical expression for $H$ that we now give. In general, the orthogonal projection onto the range of an injective transformation $T$ is $T{(T^{\ast }T)}^{-1}{T}^{\ast }$. Applying this to $T=A^{\ast }B$, we have

$H$ $=A^{\ast }B{\left(B^{\ast }A{A}^{\ast }B\right)}^{-1}{B}^{\ast }A$

$=A^{\ast }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}{B}^{\ast }A.$

We see that $HZ=A^{\ast }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}Y$.

We have proved that conditioning on $Y$ and $HZ$ are equivalent, and so:

$$𝔼[Z\mid Y]=𝔼[Z\mid HZ]=𝔼[HZ+(I-H)Z\mid HZ]=HZ+0,$$

and

$\mathrm{Var}[Z\mid Y]=\mathrm{Var}[Z\mid HZ]$ $=\mathrm{Var}[HZ+(I-H)Z\mid HZ]$

$=0+\mathrm{Var}[(I-H)Z]$

$=𝔼\left[(I-H)Z{Z}^{\ast }{(I-H)}^{\ast }\right]$

$=(I-H){(I-H)}^{\ast }$

$=I-H-H^{\ast }+H{H}^{\ast }=I-H,$

using the defining property $H^2=H=H^{\ast }$ of orthogonal projections.

Now we express the result in terms of $X$, and remove the dependence on the transformation $A$ (which is not uniquely defined from the covariance matrix):

$$𝔼[X\mid Y]=A𝔼[Z\mid Y]=AHZ=\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}Y$$

and

$$\mathrm{Var}[X\mid Y]=A\mathrm{Var}[Z\mid Y]A^{\ast }=A{A}^{\ast }-AH{A}^{\ast }=\mathrm{\Sigma }-\mathrm{\Sigma }B{(B^{\ast }\mathrm{\Sigma }B)}^{-1}{B}^{\ast }\mathrm{\Sigma }.$$

Of course, the conditional distribution of $X$ given $Y$ is the same as that of $(I-H)Z$, which is multi-variate normal.

The formula in the statement of this theorem, for the single-dimensional case, follows from substituting in $\mathrm{Var}[Y]=\mathrm{Var}[B^{\ast }X]=B^{\ast }\mathrm{\Sigma }B$. The formula for when $X$ does not have zero mean follows from applying the base case to the shifted variable $X-𝔼[X]$. ∎

Title	conditional distribution of multi-variate normal variable
Canonical name	ConditionalDistributionOfMultivariateNormalVariable
Date of creation	2013-03-22 18:39:09
Last modified on	2013-03-22 18:39:09
Owner	stevecheng (10074)
Last modified by	stevecheng (10074)
Numerical id	5
Author	stevecheng (10074)
Entry type	Theorem
Classification	msc 62E15
Classification	msc 60E05