exponential family
A probability (density^{}) function^{} ${f}_{X}(x\mid \theta )$ given a parameter $\theta $ is said to belong to the (one parameter) exponential family of distributions^{} if it can be written in one of the following two equivalent^{} forms:

1.
$a(x)b(\theta )\mathrm{exp}\left[c(x)d(\theta )\right]$

2.
$\mathrm{exp}\left[a(x)+b(\theta )+c(x)d(\theta )\right]$
where $a,b,c,d$ are known functions. If $c(x)=x$, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function $d(\theta )$ is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.
Examples:

•
The normal distribution^{}, $N(\mu ,{\sigma}^{2})$, treating ${\sigma}^{2}$ as a nuisance parameter, belongs to the exponential family. To see this, take the natural logarithm^{} of $N(\mu ,{\sigma}^{2})$ to get
$$\frac{1}{2}\mathrm{ln}(2\pi {\sigma}^{2})\frac{1}{2{\sigma}^{2}}{(x\mu )}^{2}$$ Rearrange the above expression and we have
$$\frac{x\mu}{{\sigma}^{2}}\frac{{\mu}^{2}}{2{\sigma}^{2}}\frac{1}{2}\left[\frac{{x}^{2}}{{\sigma}^{2}}+\mathrm{ln}(2\pi {\sigma}^{2})\right]$$ Set $c(x)=x$, $d(\mu )=\mu /{\sigma}^{2}$, $b(\mu )={\mu}^{2}/(2{\sigma}^{2})$, and $a(x)=1/2\left[{x}^{2}/{\sigma}^{2}+\mathrm{ln}(2\pi {\sigma}^{2})\right]$. Then we see that $N(\mu ,{\sigma}^{2})$ does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is $d(\mu )=\mu /{\sigma}^{2}$.

•
Similarly, the Poisson, binomial, Gamma, and inverse Gaussian distributions all belong to the exponential family and they are all in canonical form.

•
Lognormal and Weibull distributions^{} also belong to the exponential family but they are not in canonical form.
Remarks

•
If the p.d.f of a random variable^{} $X$ belongs to an exponential family, and it is expressed in the second of the two above forms, then
$$\mathrm{E}[c(X)]=\frac{{b}^{\prime}(\theta )}{{d}^{\prime}(\theta )},$$ (1) and
$$\mathrm{Var}[c(X)]=\frac{{d}^{\prime \prime}(\theta ){b}^{\prime}(\theta ){d}^{\prime}(\theta ){b}^{\prime \prime}(\theta )}{{d}^{\prime}{(\theta )}^{3}},$$ (2) provided that functions $b$ and $d$ are appropriately conditioned.

•
Given a member from the exponential family of distributions, we have $\mathrm{E}[U]=0$ and $I=\mathrm{E}[{U}^{\prime}]$, where $U$ is the score function^{} and $I$ the Fisher information^{}. To see this, first observe that the loglikelihood function^{} from a member of the exponential family of distributions is given by
$$\mathrm{\ell}(\theta \mid x)=a(x)+b(\theta )+c(x)d(\theta ),$$ and hence the score function is
$$U(\theta )={b}^{\prime}(\theta )+c(X){d}^{\prime}(\theta ).$$ From (1), $\mathrm{E}[U]=0$. Next, we obtain the Fisher information $I$. By definition, we have
$I$ $=$ $\mathrm{E}[{U}^{2}]\mathrm{E}{[U]}^{2}$ $=$ $\mathrm{E}[{U}^{2}]$ $=$ ${d}^{\prime}{(\theta )}^{2}\mathrm{Var}[c(X)]$ $=$ $\frac{{d}^{\prime \prime}(\theta ){b}^{\prime}(\theta ){d}^{\prime}(\theta ){b}^{\prime \prime}(\theta )}{{d}^{\prime}(\theta )}$ On the other hand,
$$\frac{\partial U}{\partial \theta}={b}^{\prime \prime}(\theta )+c(X){d}^{\prime \prime}(\theta )$$ so
$\mathrm{E}\left[{\displaystyle \frac{\partial U}{\partial \theta}}\right]$ $=$ ${b}^{\prime \prime}(\theta )+\mathrm{E}[c(X)]{d}^{\prime \prime}(\theta )$ $=$ ${b}^{\prime \prime}(\theta ){\displaystyle \frac{{b}^{\prime}(\theta )}{{d}^{\prime}(\theta )}}{d}^{\prime \prime}(\theta )$ $=$ $\frac{{b}^{\prime \prime}(\theta ){d}^{\prime}(\theta ){b}^{\prime}(\theta ){d}^{\prime \prime}(\theta )}{{d}^{\prime}(\theta )}$ $=$ $I$ 
•
For example, for a Poisson distribution^{}
$${f}_{X}(x\mid \theta )=\frac{{\theta}^{x}{e}^{\theta}}{x!},$$ the natural parameter $d(\theta )$ is $\mathrm{ln}\theta $ and $b(\theta )=\theta $. $c(x)=x$ since Poisson is in canonical form. Then
$$U(\theta )=1+\frac{X}{\theta}\text{and}I=\mathrm{E}\left[\frac{X}{{\theta}^{2}}\right]=\frac{1}{\theta}$$ as expected.
Title  exponential family 

Canonical name  ExponentialFamily 
Date of creation  20130322 14:30:08 
Last modified on  20130322 14:30:08 
Owner  CWoo (3771) 
Last modified by  CWoo (3771) 
Numerical id  7 
Author  CWoo (3771) 
Entry type  Definition 
Classification  msc 62J12 
Defines  canonical exponential family 
Defines  nuisance parameter 
Defines  natural parameter 