PlanetMath: exponential family

(more info)

Math for the people, by the people.

Main Menu

sections Encyclopædia
Papers
Books
Expositions

meta Requests (200)
Orphanage
Unclass'd
Unproven (490)
Corrections (6)
Classification

talkback Polls
Forums
Feedback
Bug Reports

downloads Snapshots
PM Book

information News
Docs
Wiki
ChangeLog
TODO List
Legalese
About

exponential family

(Definition)

A probability (density) function $f_X(x\mid\theta)$ given a parameter $\theta$ is said to belong to the (one parameter) exponential family of distributions if it can be written in one of the following two equivalent forms:

$a(x)b(\theta)\operatorname{exp}\big[ c(x)d(\theta)\big ]$
$\operatorname{exp}\big[ a(x)+b(\theta)+c(x)d(\theta) \big]$

where

are known functions. If

, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function $d(\theta)$ is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.

Examples:

The normal distribution, $N(\mu,\sigma^2)$ , treating $\sigma^2$ as a nuisance parameter, belongs to the exponential family. To see this, take the natural logarithm of $N(\mu,\sigma^2)$ to get
$\displaystyle -\frac{1}{2}\operatorname{ln}(2\pi\sigma^2)-\frac{1}{2\sigma^2}(x-\mu)^2$
Rearrange the above expression and we have
$\displaystyle \frac{x\mu}{\sigma^2}-\frac{\mu^2}{2\sigma^2}-\frac{1}{2}\Big[\frac{x^2}{\sigma^2}+\operatorname{ln}(2\pi\sigma^2)\Big]$
Set , $d(\mu)=\mu/\sigma^2$ , $b(\mu)=-\mu^2/(2\sigma^2)$ , and $a(x)=-1/2\big[x^2/\sigma^2+\operatorname{ln}(2\pi\sigma^2)\big]$ . Then we see that $N(\mu,\sigma^2)$ does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is $d(\mu)=\mu/\sigma^2$ .
Similarly, the Poisson, binomial, Gamma, and inverse Gaussian distributions all belong to the exponential family and they are all in canonical form.
Lognormal and Weibull distributions also belong to the exponential family but they are not in canonical form.

Remarks

If the p.d.f of a random variable belongs to an exponential family, and it is expressed in the second of the two above forms, then

$\displaystyle \operatorname{E}[c(X)]=-\frac{b'(\theta)}{d'(\theta)},$ (1)

and

$\displaystyle \operatorname{Var}[c(X)]=\frac{d''(\theta)b'(\theta)-d'(\theta)b''(\theta)}{d'(\theta)^3},$ (2)

provided that functions and are appropriately conditioned.
Given a member from the exponential family of distributions, we have $\operatorname{E}[U]=0$ and $I=-\operatorname{E}[U']$ , where is the score function and the Fisher information. To see this, first observe that the log-likelihood function from a member of the exponential family of distributions is given by
$\displaystyle \ell(\theta\mid x)=a(x)+b(\theta)+c(x)d(\theta),$
and hence the score function is
$\displaystyle U(\theta)=b'(\theta)+c(X)d'(\theta).$
From (1), $\operatorname{E}[U]=0$ . Next, we obtain the Fisher information . By definition, we have

$\displaystyle I$ $\displaystyle =$ $\displaystyle \operatorname{E}[U^2]-\operatorname{E}[U]^2$

$\displaystyle =$ $\displaystyle \operatorname{E}[U^2]$

$\displaystyle =$ $\displaystyle d'(\theta)^2\operatorname{Var}[c(X)]$

$\displaystyle =$ $\displaystyle \frac{d''(\theta)b'(\theta)-d'(\theta)b''(\theta)}{d'(\theta)}$

On the other hand,
$\displaystyle \frac{\partial U}{\partial\theta}=b''(\theta)+c(X)d''(\theta)$
so

$\displaystyle \operatorname{E}\Big[\frac{\partial U}{\partial\theta}\Big]$ $\displaystyle =$ $\displaystyle b''(\theta)+\operatorname{E}[c(X)]d''(\theta)$

$\displaystyle =$ $\displaystyle b''(\theta)-\frac{b'(\theta)}{d'(\theta)}d''(\theta)$

$\displaystyle =$ $\displaystyle \frac{b''(\theta)d'(\theta)-b'(\theta)d''(\theta)}{d'(\theta)}$

$\displaystyle =$ $\displaystyle -I$
For example, for a Poisson distribution
$\displaystyle f_X(x\mid\theta) = \frac{\theta^x e^{-\theta}}{x!},$
the natural parameter $d(\theta)$ is $\operatorname{ln}\theta$ and $b(\theta)=-\theta$ . since Poisson is in canonical form. Then
$\displaystyle U(\theta)=-1+\frac{X}{\theta}$ and $\displaystyle I=-\operatorname{E}\Big[\frac{-X}{\theta^2}\Big]=\frac{1}{\theta}$
as expected.