|
|
|
Viewing Version
4
of
'Fisher information matrix'
|
[ view 'Fisher information matrix'
|
back to history
]
| Title of object: |
Fisher information matrix |
| Canonical Name: |
FisherInformationMatrix |
| Type: |
Definition |
| Created on: |
2004-07-27 20:44:26 |
| Modified on: |
2005-02-28 13:34:42 |
| Classification: |
msc:62A01, msc:62B10, msc:62H99 |
| Defines: |
Fisher information, information, Cramer-Rao inequality, Cramer-Rao lower bound |
| Synonyms: |
Fisher information matrix=information matrix |
Revision comment (for changes between this and next version):
| Changes for correction #7792 ('Typo in $\partial U/\partial\boldsymbol{\theta}$'). |
Preamble:
% this is the default PlanetMath preamble. as your knowledge
% of TeX increases, you will probably want to edit this, but
% it should be fine as is for beginners.
% almost certainly you want these
\usepackage{amssymb,amscd}
\usepackage{amsmath}
\usepackage{amsfonts}
% used for TeXing text within eps files
%\usepackage{psfrag}
% need this for including graphics (\includegraphics)
%\usepackage{graphicx}
% for neatly defining theorems and propositions
%\usepackage{amsthm}
% making logically defined graphics
%\usepackage{xypic}
% there are many more packages, add them here as you need them
% define commands here |
Content:
\PMlinkescapeword{model}
Given a statistical model $\lbrace f_\textbf{X}(\boldsymbol{x}\mid\boldsymbol{\theta})\rbrace$ of a random vector $\textbf{X}$, the \emph{Fisher information matrix}, $I$, is the variance of the score function $U$. So,
$$I=\operatorname{Var}(U).$$
If there is only one parameter involved, then $I$ is simply called the Fisher information or information of $f_\textbf{X}(\boldsymbol{x}\mid\theta)$.
\textbf{Remarks}
\begin{itemize}
\item If $f_\textbf{X}(\boldsymbol{x}\mid\boldsymbol{\theta})$ belongs to the exponential family, $I=\operatorname{E}(U^{\operatorname{T}}U)$. Furthermore, with some regularity conditions imposed, $I=-\operatorname{E}(\partial U/\partial\boldsymbol{\theta})$.
\item As an example, the normal distribution, $N(\mu,\sigma^2)$, belongs to the exponential family and its log-likelihood function $\ell(\boldsymbol{\theta}\mid x)$ is $$-\frac{1}{2}\operatorname{ln}(2\pi\sigma^2)-\frac{(x-\mu)^2}{2\sigma^2},$$ where $\boldsymbol{\theta}=(\mu,\sigma^2)$. Then the score function $U(\boldsymbol{\theta})$ is given by
$$(\frac{x-\mu}{\sigma^2},\frac{(x-\mu)^2}{2\sigma^4}-\frac{1}{2\sigma^2}).$$
Taking the derivative with respect to $\boldsymbol{\theta}$, we have
$$\partial U/\partial\boldsymbol{\theta}=
\begin{pmatrix}
-1/\sigma^2 & -(x-\mu)/\sigma^4 \\
-(x-\mu)(\sigma^4) & 1/(2\sigma^4)-(x-\mu)^2/(4\sigma^6)
\end{pmatrix}.$$
Therefore, the Fisher information matrix $I$ is
$$-\operatorname{E}(\partial U/\partial\boldsymbol{\theta})=
\begin{pmatrix}
1/\sigma^2 & 0 \\
0 & 1/(2\sigma^4)
\end{pmatrix}.$$
\item Now, in linear regression model with constant variance $\sigma^2$, it can be shown that the Fisher information matrix $I$ is
$$\frac{1}{\sigma^2}\textbf{X}^{\operatorname{T}}\textbf{X},$$
where $\textbf{X}$ is the design matrix of the regression model.
\item In general, the Fisher information meansure how much ``information'' is known about a parameter $\theta$. If $T$ is an unbiased estimator of $\theta$, it can be shown that
$$\operatorname{Var}[T(X)]\ge\frac{1}{I(\theta)}$$
This is known as the Cramer-Rao inequality, and the number $1/I(\theta)$ is known as the Cramer-Rao lower bound. The samller the variance of the estimate of $\theta$, the more information we have on $\theta$. If there is more than one parameter, the above can be generalized by saying that
$$\operatorname{Var}[T(X)]-I(\boldsymbol{\theta})^{-1}$$ is positive semidefinite, where $I$ is the Fisher information matrix.
\end{itemize} |
|
|
|
|
|
