likelihood function

Let X=($X_1,\mathrm{\dots },X_n$) be a random vector and

$$\{f_{𝐗}(𝒙\mid 𝜽):𝜽\in \mathrm{\Theta }\}$$

a statistical model parametrized by $𝜽=({\theta }_1,\mathrm{\dots },{\theta }_k)$, the parameter vector in the parameter space $\mathrm{\Theta }$. The likelihood function is a map $L:\mathrm{\Theta }\to ℝ$ given by

$$L(𝜽\mid 𝒙)=f_{𝐗}(𝒙\mid 𝜽).$$

In other words, the likelikhood function is functionally the same in form as a probability density function. However, the emphasis is changed from the $𝒙$ to the $𝜽$. The pdf is a function of the $x$’s while holding the parameters $\theta$’s constant, $L$ is a function of the parameters $\theta$’s, while holding the $x$’s constant.

When there is no confusion, $L(𝜽\mid 𝒙)$ is abbreviated to be $L(𝜽)$.

The parameter vector $\widehat{𝜽}$ such that $L(\widehat{𝜽})\ge L(𝜽)$ for all $𝜽\in \mathrm{\Theta }$ is called a maximum likelihood estimate, or MLE, of $𝜽$.

Many of the density functions are exponential in nature, it is therefore easier to compute the MLE of a likelihood function $L$ by finding the maximum of the natural log of $L$, known as the log-likelihood function:

$$\mathrm{\ell }(𝜽\mid 𝒙)=\ln (L(𝜽\mid 𝒙))$$

due to the monotonicity of the log function.

Examples:

1. A coin is tossed $n$ times and $m$ heads are observed. Assume that the probability of a head after one toss is $\pi$. What is the MLE of $\pi$?

   Solution: Define the outcome of a toss be 0 if a tail is observed and 1 if a head is observed. Next, let $X_i$ be the outcome of the $i$th toss. For any single toss, the density function is ${\pi }^x{(1-\pi )}^{1-x}$ where $x\in \{0,1\}$. Assume that the tosses are independent events, then the joint probability density is

   $$f_{𝐗}(𝒙\mid \pi )=\left(\genfrac{}{}{0pt}{}{n}{\mathrm{\Sigma }{x}_i}\right){\pi }^{\mathrm{\Sigma }{x}_i}{(1-\pi )}^{\mathrm{\Sigma }(1-x_i)}=\left(\genfrac{}{}{0pt}{}{n}{m}\right){\pi }^m{(1-\pi )}^{n-m},$$

   which is also the likelihood function $L(\pi )$. Therefore, the log-likelihood function has the form

   $$\mathrm{\ell }(\pi \mid 𝒙)=\mathrm{\ell }(\pi )=\ln \left(\genfrac{}{}{0pt}{}{n}{m}\right)+m\ln (\pi )+(n-m)\ln (1-\pi ).$$

   Using standard calculus, we get that the MLE of $\pi$ is

   $$\widehat{\pi }=\frac{m}{n}=\overline{x}.$$

2. Suppose a sample of $n$ data points $X_i$ are collected. Assume that the $X_i\sim N(\mu ,{\sigma }^2)$ and the $X_i$’s are independent of each other. What is the MLE of the parameter vector $𝜽=(\mu ,{\sigma }^2)$?

   Solution: The joint pdf of the $X_i$, and hence the likelihood function, is

   $$L(𝜽\mid 𝒙)=\frac{1}{{\sigma }^n{(2\pi )}^{n/2}}\exp (-\frac{\mathrm{\Sigma }{(x_i-\mu )}^2}{2{\sigma }^2}).$$

   The log-likelihood function is

   $$\mathrm{\ell }(𝜽\mid 𝒙)=-\frac{\mathrm{\Sigma }{(x_i-\mu )}^2}{2{\sigma }^2}-\frac{n}{2}\ln ({\sigma }^2)-\frac{n}{2}\ln (2\pi ).$$

   Taking the first derivative (gradient), we get

   $$\frac{\partial \mathrm{\ell }}{\partial 𝜽}=(\frac{\mathrm{\Sigma }(x_i-\mu )}{{\sigma }^2},\frac{\mathrm{\Sigma }{(x_i-\mu )}^2}{2{\sigma }^4}-\frac{n}{2{\sigma }^2}).$$

   Setting

   $$\frac{\partial \mathrm{\ell }}{\partial 𝜽}=\mathrm{𝟎}\text{ See score function}$$

   and solve for $𝜽=(\mu ,{\sigma }^2)$ we have

   $$\widehat{𝜽}=(\widehat{\mu },{\widehat{\sigma }}^2)=(\overline{x},\frac{n-1}{n}{s}^2),$$

   where $\overline{x}=\mathrm{\Sigma }{x}_i/n$ is the sample mean and $s^2=\mathrm{\Sigma }{(x_i-\overline{x})}^2/(n-1)$ is the sample variance. Finally, we verify that $\widehat{𝜽}$ is indeed the MLE of $𝜽$ by checking the negativity of the 2nd derivatives (for each parameter).

Title	likelihood function
Canonical name	LikelihoodFunction
Date of creation	2013-03-22 14:27:58
Last modified on	2013-03-22 14:27:58
Owner	CWoo (3771)
Last modified by	CWoo (3771)
Numerical id	13
Author	CWoo (3771)
Entry type	Definition
Classification	msc 62A01
Synonym	likelihood statistic
Synonym	likelihood
Defines	maximum likelihood estimate
Defines	MLE
Defines	log-likelihood function