# exponential family

1. 1.

$a(x)b(\theta)\operatorname{exp}\big{[}c(x)d(\theta)\big{]}$

2. 2.

$\operatorname{exp}\big{[}a(x)+b(\theta)+c(x)d(\theta)\big{]}$

where $a,b,c,d$ are known functions. If $c(x)=x$, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function $d(\theta)$ is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.

Examples:

Remarks

• If the p.d.f of a random variable  $X$ belongs to an exponential family, and it is expressed in the second of the two above forms, then

 $\operatorname{E}[c(X)]=-\frac{b^{\prime}(\theta)}{d^{\prime}(\theta)},$ (1)

and

 $\operatorname{Var}[c(X)]=\frac{d^{\prime\prime}(\theta)b^{\prime}(\theta)-d^{% \prime}(\theta)b^{\prime\prime}(\theta)}{d^{\prime}(\theta)^{3}},$ (2)

provided that functions $b$ and $d$ are appropriately conditioned.

• Given a member from the exponential family of distributions, we have $\operatorname{E}[U]=0$ and $I=-\operatorname{E}[U^{\prime}]$, where $U$ is the score function  and $I$ the Fisher information  . To see this, first observe that the log-likelihood function  from a member of the exponential family of distributions is given by

 $\ell(\theta\mid x)=a(x)+b(\theta)+c(x)d(\theta),$

and hence the score function is

 $U(\theta)=b^{\prime}(\theta)+c(X)d^{\prime}(\theta).$

From (1), $\operatorname{E}[U]=0$. Next, we obtain the Fisher information $I$. By definition, we have

 $\displaystyle I$ $\displaystyle=$ $\displaystyle\operatorname{E}[U^{2}]-\operatorname{E}[U]^{2}$ $\displaystyle=$ $\displaystyle\operatorname{E}[U^{2}]$ $\displaystyle=$ $\displaystyle d^{\prime}(\theta)^{2}\operatorname{Var}[c(X)]$ $\displaystyle=$ $\displaystyle\frac{d^{\prime\prime}(\theta)b^{\prime}(\theta)-d^{\prime}(% \theta)b^{\prime\prime}(\theta)}{d^{\prime}(\theta)}$

On the other hand,

 $\frac{\partial U}{\partial\theta}=b^{\prime\prime}(\theta)+c(X)d^{\prime\prime% }(\theta)$

so

 $\displaystyle\operatorname{E}\Big{[}\frac{\partial U}{\partial\theta}\Big{]}$ $\displaystyle=$ $\displaystyle b^{\prime\prime}(\theta)+\operatorname{E}[c(X)]d^{\prime\prime}(\theta)$ $\displaystyle=$ $\displaystyle b^{\prime\prime}(\theta)-\frac{b^{\prime}(\theta)}{d^{\prime}(% \theta)}d^{\prime\prime}(\theta)$ $\displaystyle=$ $\displaystyle\frac{b^{\prime\prime}(\theta)d^{\prime}(\theta)-b^{\prime}(% \theta)d^{\prime\prime}(\theta)}{d^{\prime}(\theta)}$ $\displaystyle=$ $\displaystyle-I$
•  $f_{X}(x\mid\theta)=\frac{\theta^{x}e^{-\theta}}{x!},$

the natural parameter $d(\theta)$ is $\operatorname{ln}\theta$ and $b(\theta)=-\theta$. $c(x)=x$ since Poisson is in canonical form. Then

 $U(\theta)=-1+\frac{X}{\theta}\mbox{ and }I=-\operatorname{E}\Big{[}\frac{-X}{% \theta^{2}}\Big{]}=\frac{1}{\theta}$

as expected.

Title exponential family ExponentialFamily 2013-03-22 14:30:08 2013-03-22 14:30:08 CWoo (3771) CWoo (3771) 7 CWoo (3771) Definition msc 62J12 canonical exponential family nuisance parameter natural parameter