# exponential family

A probability (density) function $f_{X}(x\mid\theta)$ given a parameter $\theta$ is said to belong to the (one parameter) exponential family of distributions if it can be written in one of the following two equivalent forms:

1. 1.

$a(x)b(\theta)\operatorname{exp}\big{[}c(x)d(\theta)\big{]}$

2. 2.

$\operatorname{exp}\big{[}a(x)+b(\theta)+c(x)d(\theta)\big{]}$

where $a,b,c,d$ are known functions. If $c(x)=x$, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function $d(\theta)$ is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.

Examples:

• The normal distribution, $N(\mu,\sigma^{2})$, treating $\sigma^{2}$ as a nuisance parameter, belongs to the exponential family. To see this, take the natural logarithm of $N(\mu,\sigma^{2})$ to get

 $-\frac{1}{2}\operatorname{ln}(2\pi\sigma^{2})-\frac{1}{2\sigma^{2}}(x-\mu)^{2}$

Rearrange the above expression and we have

 $\frac{x\mu}{\sigma^{2}}-\frac{\mu^{2}}{2\sigma^{2}}-\frac{1}{2}\Big{[}\frac{x^% {2}}{\sigma^{2}}+\operatorname{ln}(2\pi\sigma^{2})\Big{]}$

Set $c(x)=x$, $d(\mu)=\mu/\sigma^{2}$, $b(\mu)=-\mu^{2}/(2\sigma^{2})$, and $a(x)=-1/2\big{[}x^{2}/\sigma^{2}+\operatorname{ln}(2\pi\sigma^{2})\big{]}$. Then we see that $N(\mu,\sigma^{2})$ does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is $d(\mu)=\mu/\sigma^{2}$.

• Similarly, the Poisson, binomial, Gamma, and inverse Gaussian distributions all belong to the exponential family and they are all in canonical form.

• Lognormal and Weibull distributions also belong to the exponential family but they are not in canonical form.

Remarks

• If the p.d.f of a random variable $X$ belongs to an exponential family, and it is expressed in the second of the two above forms, then

 $\operatorname{E}[c(X)]=-\frac{b^{\prime}(\theta)}{d^{\prime}(\theta)},$ (1)

and

 $\operatorname{Var}[c(X)]=\frac{d^{\prime\prime}(\theta)b^{\prime}(\theta)-d^{% \prime}(\theta)b^{\prime\prime}(\theta)}{d^{\prime}(\theta)^{3}},$ (2)

provided that functions $b$ and $d$ are appropriately conditioned.

• Given a member from the exponential family of distributions, we have $\operatorname{E}[U]=0$ and $I=-\operatorname{E}[U^{\prime}]$, where $U$ is the score function and $I$ the Fisher information. To see this, first observe that the log-likelihood function from a member of the exponential family of distributions is given by

 $\ell(\theta\mid x)=a(x)+b(\theta)+c(x)d(\theta),$

and hence the score function is

 $U(\theta)=b^{\prime}(\theta)+c(X)d^{\prime}(\theta).$

From (1), $\operatorname{E}[U]=0$. Next, we obtain the Fisher information $I$. By definition, we have

 $\displaystyle I$ $\displaystyle=$ $\displaystyle\operatorname{E}[U^{2}]-\operatorname{E}[U]^{2}$ $\displaystyle=$ $\displaystyle\operatorname{E}[U^{2}]$ $\displaystyle=$ $\displaystyle d^{\prime}(\theta)^{2}\operatorname{Var}[c(X)]$ $\displaystyle=$ $\displaystyle\frac{d^{\prime\prime}(\theta)b^{\prime}(\theta)-d^{\prime}(% \theta)b^{\prime\prime}(\theta)}{d^{\prime}(\theta)}$

On the other hand,

 $\frac{\partial U}{\partial\theta}=b^{\prime\prime}(\theta)+c(X)d^{\prime\prime% }(\theta)$

so

 $\displaystyle\operatorname{E}\Big{[}\frac{\partial U}{\partial\theta}\Big{]}$ $\displaystyle=$ $\displaystyle b^{\prime\prime}(\theta)+\operatorname{E}[c(X)]d^{\prime\prime}(\theta)$ $\displaystyle=$ $\displaystyle b^{\prime\prime}(\theta)-\frac{b^{\prime}(\theta)}{d^{\prime}(% \theta)}d^{\prime\prime}(\theta)$ $\displaystyle=$ $\displaystyle\frac{b^{\prime\prime}(\theta)d^{\prime}(\theta)-b^{\prime}(% \theta)d^{\prime\prime}(\theta)}{d^{\prime}(\theta)}$ $\displaystyle=$ $\displaystyle-I$
• For example, for a Poisson distribution

 $f_{X}(x\mid\theta)=\frac{\theta^{x}e^{-\theta}}{x!},$

the natural parameter $d(\theta)$ is $\operatorname{ln}\theta$ and $b(\theta)=-\theta$. $c(x)=x$ since Poisson is in canonical form. Then

 $U(\theta)=-1+\frac{X}{\theta}\mbox{ and }I=-\operatorname{E}\Big{[}\frac{-X}{% \theta^{2}}\Big{]}=\frac{1}{\theta}$

as expected.

Title exponential family ExponentialFamily 2013-03-22 14:30:08 2013-03-22 14:30:08 CWoo (3771) CWoo (3771) 7 CWoo (3771) Definition msc 62J12 canonical exponential family nuisance parameter natural parameter