You are here
Homeexponential family
Primary tabs
exponential family
A probability (density) function $f_{X}(x\mid\theta)$ given a parameter $\theta$ is said to belong to the (one parameter) exponential family of distributions if it can be written in one of the following two equivalent forms:
1. $a(x)b(\theta)\operatorname{exp}\big[c(x)d(\theta)\big]$
2. $\operatorname{exp}\big[a(x)+b(\theta)+c(x)d(\theta)\big]$
where $a,b,c,d$ are known functions. If $c(x)=x$, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function $d(\theta)$ is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.
Examples:

The normal distribution, $N(\mu,\sigma^{2})$, treating $\sigma^{2}$ as a nuisance parameter, belongs to the exponential family. To see this, take the natural logarithm of $N(\mu,\sigma^{2})$ to get
$\frac{1}{2}\operatorname{ln}(2\pi\sigma^{2})\frac{1}{2\sigma^{2}}(x\mu)^{2}$ Rearrange the above expression and we have
$\frac{x\mu}{\sigma^{2}}\frac{\mu^{2}}{2\sigma^{2}}\frac{1}{2}\Big[\frac{x^{2% }}{\sigma^{2}}+\operatorname{ln}(2\pi\sigma^{2})\Big]$ Set $c(x)=x$, $d(\mu)=\mu/\sigma^{2}$, $b(\mu)=\mu^{2}/(2\sigma^{2})$, and $a(x)=1/2\big[x^{2}/\sigma^{2}+\operatorname{ln}(2\pi\sigma^{2})\big]$. Then we see that $N(\mu,\sigma^{2})$ does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is $d(\mu)=\mu/\sigma^{2}$.

Lognormal and Weibull distributions also belong to the exponential family but they are not in canonical form.
Remarks

If the p.d.f of a random variable $X$ belongs to an exponential family, and it is expressed in the second of the two above forms, then
$\operatorname{E}[c(X)]=\frac{b^{{\prime}}(\theta)}{d^{{\prime}}(\theta)},$ (1) and
$\operatorname{Var}[c(X)]=\frac{d^{{\prime\prime}}(\theta)b^{{\prime}}(\theta)% d^{{\prime}}(\theta)b^{{\prime\prime}}(\theta)}{d^{{\prime}}(\theta)^{3}},$ (2) provided that functions $b$ and $d$ are appropriately conditioned.

Given a member from the exponential family of distributions, we have $\operatorname{E}[U]=0$ and $I=\operatorname{E}[U^{{\prime}}]$, where $U$ is the score function and $I$ the Fisher information. To see this, first observe that the loglikelihood function from a member of the exponential family of distributions is given by
$\ell(\theta\mid x)=a(x)+b(\theta)+c(x)d(\theta),$ and hence the score function is
$U(\theta)=b^{{\prime}}(\theta)+c(X)d^{{\prime}}(\theta).$ From (1), $\operatorname{E}[U]=0$. Next, we obtain the Fisher information $I$. By definition, we have
$\displaystyle I$ $\displaystyle=$ $\displaystyle\operatorname{E}[U^{2}]\operatorname{E}[U]^{2}$ $\displaystyle=$ $\displaystyle\operatorname{E}[U^{2}]$ $\displaystyle=$ $\displaystyle d^{{\prime}}(\theta)^{2}\operatorname{Var}[c(X)]$ $\displaystyle=$ $\displaystyle\frac{d^{{\prime\prime}}(\theta)b^{{\prime}}(\theta)d^{{\prime}}% (\theta)b^{{\prime\prime}}(\theta)}{d^{{\prime}}(\theta)}$ On the other hand,
$\frac{\partial U}{\partial\theta}=b^{{\prime\prime}}(\theta)+c(X)d^{{\prime% \prime}}(\theta)$ so
$\displaystyle\operatorname{E}\Big[\frac{\partial U}{\partial\theta}\Big]$ $\displaystyle=$ $\displaystyle b^{{\prime\prime}}(\theta)+\operatorname{E}[c(X)]d^{{\prime% \prime}}(\theta)$ $\displaystyle=$ $\displaystyle b^{{\prime\prime}}(\theta)\frac{b^{{\prime}}(\theta)}{d^{{% \prime}}(\theta)}d^{{\prime\prime}}(\theta)$ $\displaystyle=$ $\displaystyle\frac{b^{{\prime\prime}}(\theta)d^{{\prime}}(\theta)b^{{\prime}}% (\theta)d^{{\prime\prime}}(\theta)}{d^{{\prime}}(\theta)}$ $\displaystyle=$ $\displaystyleI$ 
For example, for a Poisson distribution
$f_{X}(x\mid\theta)=\frac{\theta^{x}e^{{\theta}}}{x!},$ the natural parameter $d(\theta)$ is $\operatorname{ln}\theta$ and $b(\theta)=\theta$. $c(x)=x$ since Poisson is in canonical form. Then
$U(\theta)=1+\frac{X}{\theta}\mbox{ and }I=\operatorname{E}\Big[\frac{X}{% \theta^{2}}\Big]=\frac{1}{\theta}$ as expected.
Mathematics Subject Classification
62J12 no label found Forums
 Planetary Bugs
 HS/Secondary
 University/Tertiary
 Graduate/Advanced
 Industry/Practice
 Research Topics
 LaTeX help
 Math Comptetitions
 Math History
 Math Humor
 PlanetMath Comments
 PlanetMath System Updates and News
 PlanetMath help
 PlanetMath.ORG
 Strategic Communications Development
 The Math Pub
 Testing messages (ignore)
 Other useful stuff
 Corrections
Comments
From (1), E(U)=0?
I'm confused by the E(U)=0 derivation, in particular, which is the equation (1)?