PlanetMath: likelihood function

(more info)

Math for the people, by the people.

Main Menu

sections Encyclopædia
Papers
Books
Expositions

meta Requests (200)
Orphanage
Unclass'd
Unproven (490)
Corrections (6)
Classification

talkback Polls
Forums
Feedback
Bug Reports

downloads Snapshots
PM Book

information News
Docs
Wiki
ChangeLog
TODO List
Legalese
About

Entry average rating: No information on entry rating

likelihood function

(Definition)

Let X=( $X_1,\ldots,X_n$ ) be a random vector and

$\displaystyle \lbrace f_{\mathbf{X}}(\boldsymbol{x}\mid\boldsymbol{\theta}) : \boldsymbol{\theta} \in \Theta \rbrace$

a statistical model parametrized by $\boldsymbol{\theta}=(\theta_1,\ldots,\theta_k)$ , the parameter vector in the parameter space $\Theta$ . The likelihood function is a map $L: \Theta \to \mathbb{R}$ given by

$\displaystyle L(\boldsymbol{\theta}\mid\boldsymbol{x}) = f_{\mathbf{X}}(\boldsymbol{x}\mid\boldsymbol{\theta}).$

In other words, the likelikhood function is functionally the same in form as a probability density function. However, the emphasis is changed from the $\boldsymbol{x}$ to the $\boldsymbol{\theta}$ . The pdf is a function of the

's while holding the parameters $\theta$ 's constant,

is a function of the parameters $\theta$ 's, while holding the

's constant.

When there is no confusion, $L(\boldsymbol{\theta}\mid\boldsymbol{x})$ is abbreviated to be $L(\boldsymbol{\theta})$ .

The parameter vector $\hat{\boldsymbol{\theta}}$ such that $L(\hat{\boldsymbol{\theta}})\geq L(\boldsymbol{\theta})$ for all $\boldsymbol{\theta}\in\Theta$ is called a maximum likelihood estimate, or MLE, of $\boldsymbol{\theta}$ .

Many of the density functions are exponential in nature, it is therefore easier to compute the MLE of a likelihood function by finding the maximum of the natural log of , known as the log-likelihood function:

$\displaystyle \ell(\boldsymbol{\theta}\mid\boldsymbol{x}) = \operatorname{ln}(L(\boldsymbol{\theta}\mid\boldsymbol{x}))$

due to the monotonicity of the log function.

Examples:

A coin is tossed times and heads are observed. Assume that the probability of a head after one toss is $\pi$ . What is the MLE of $\pi$ ?
Solution: Define the outcome of a toss be 0 if a tail is observed and 1 if a head is observed. Next, let be the outcome of the th toss. For any single toss, the density function is $\pi^x(1-\pi)^{1-x}$ where $x\in \lbrace 0,1\rbrace$ . Assume that the tosses are independent events, then the joint probability density is

$\displaystyle f_{\mathbf{X}}(\boldsymbol{x}\mid\pi)=\binom{n}{\Sigma x_i}\pi^{\Sigma x_i}(1-\pi)^{\Sigma (1-x_i)}=\binom{n}{m}\pi^m(1-\pi)^{n-m},$
which is also the likelihood function $L(\pi)$ . Therefore, the log-likelihood function has the form
$\displaystyle \ell(\pi\mid\boldsymbol{x})=\ell(\pi)=\operatorname{ln}\binom{n}{m}+m\operatorname{ln}(\pi)+(n-m)\operatorname{ln}(1-\pi).$
Using standard calculus, we get that the MLE of $\pi$ is
$\displaystyle \hat{\pi}=\frac{m}{n}=\overline{x}.$
Suppose a sample of data points are collected. Assume that the $X_i\sim N(\mu,\sigma^2)$ and the 's are independent of each other. What is the MLE of the parameter vector $\boldsymbol{\theta}=(\mu,\sigma^2)$ ?
Solution: The joint pdf of the , and hence the likelihood function, is

$\displaystyle L(\boldsymbol{\theta}\mid\boldsymbol{x})=\frac{1}{\sigma^n(2\pi)^{n/2}}\operatorname{exp}(-\frac{\Sigma(x_i-\mu)^2}{2\sigma^2}).$
The log-likelihood function is
$\displaystyle \ell(\boldsymbol{\theta}\mid\boldsymbol{x})=-\frac{\Sigma(x_i-\mu... ...a^2}-\frac{n}{2}\operatorname{ln}(\sigma^2)-\frac{n}{2}\operatorname{ln}(2\pi).$
Taking the first derivative (gradient), we get
$\displaystyle \frac{\partial\ell}{\partial \boldsymbol{\theta}}=(\frac{\Sigma(x_i-\mu)}{\sigma^2},\frac{\Sigma(x_i-\mu)^2}{2\sigma^4}-\frac{n}{2\sigma^2}).$
Setting
$\displaystyle \frac{\partial\ell}{\partial \boldsymbol{\theta}}=\boldsymbol{0}$ See score function
and solve for $\boldsymbol{\theta}=(\mu,\sigma^2)$ we have
$\displaystyle \boldsymbol{\hat{\theta}}=(\hat{\mu},\hat{\sigma}^2)=(\overline{x},\frac{n-1}{n}s^2),$
where $\overline{x}=\Sigma x_i/n$ is the sample mean and $s^2=\Sigma (x_i-\overline{x})^2/(n-1)$ is the sample variance. Finally, we verify that $\hat{\boldsymbol{\theta}}$ is indeed the MLE of $\boldsymbol{\theta}$ by checking the negativity of the 2nd derivatives (for each parameter).