Let X=($X_1,\dots ,X_n$) be a random vector and

a statistical model parametrized by $𝜽=({\theta }_1,\dots ,{\theta }_k)$, the parameter vector in the parameter space $\Theta$. The likelihood function is a map $L:\Theta \to ℝ$ given by

In other words, the likelikhood function is functionally the same in form as a probability density function. However, the emphasis is changed from the $𝒙$ to the $𝜽$. The pdf is a function of the $x$’s while holding the parameters $\theta$’s constant, $L$ is a function of the parameters $\theta$’s, while holding the $x$’s constant.

When there is no confusion, $L(𝜽\mid 𝒙)$ is abbreviated to be $L\left(𝜽\right)$.

The parameter vector $\widehat{𝜽}$ such that $L\left(\widehat{𝜽}\right)\ge L\left(𝜽\right)$ for all $𝜽\in \Theta$ is called a maximum likelihood estimate, or MLE, of $𝜽$.

Many of the density functions are exponential in nature, it is therefore easier to compute the MLE of a likelihood function $L$ by finding the maximum of the natural log of $L$, known as the log-likelihood function:

due to the monotonicity of the log function.

Examples:

1. 1.

   A coin is tossed $n$ times and $m$ heads are observed. Assume that the probability of a head after one toss is $\pi$. What is the MLE of $\pi$?

   Solution: Define the outcome of a toss be 0 if a tail is observed and 1 if a head is observed. Next, let $X_i$ be the outcome of the $i$th toss. For any single toss, the density function is ${\pi }^x{\left(1-\pi \right)}^{1-x}$ where $x\in \{0,1\}$. Assume that the tosses are independent events, then the joint probability density is

   $f_{𝐗}(𝒙\mid \pi )=\left(\frac{n}{\Sigma x_i}\right){\pi }^{\Sigma x_i}{(1-\pi )}^{\Sigma \left(1-x_i\right)}=\left(\frac{n}{m}\right){\pi }^m{(1-\pi )}^{n-m},$

   which is also the likelihood function $L\left(\pi \right)$. Therefore, the log-likelihood function has the form

   $\ell (\pi \mid 𝒙)=\ell \left(\pi \right)=\ln \left(\frac{n}{m}\right)+m\ln \left(\pi \right)+(n-m)\ln (1-\pi ).$

   Using standard calculus, we get that the MLE of $\pi$ is

   $\hat{\pi }=\frac{m}{n}=\overline{x}.$

2. 2.

   Suppose a sample of $n$ data points $X_i$ are collected. Assume that the $X_i\sim N(\mu ,{\sigma }^2)$ and the $X_i$’s are independent of each other. What is the MLE of the parameter vector $𝜽=(\mu ,{\sigma }^2)$?

   Solution: The joint pdf of the $X_i$, and hence the likelihood function, is

   $L(𝜽\mid 𝒙)=\frac{1}{{\sigma }^n{\left(2\pi \right)}^{n/2}}\exp (-\frac{\Sigma {\left(x_i-\mu \right)}^2}{2{\sigma }^2}).$

   The log-likelihood function is

   $\ell (𝜽\mid 𝒙)=-\frac{\Sigma {\left(x_i-\mu \right)}^2}{2{\sigma }^2}-\frac{n}{2}\ln \left({\sigma }^2\right)-\frac{n}{2}\ln \left(2\pi \right).$

   Taking the first derivative (gradient), we get

   $\frac{\partial \ell }{\partial 𝜽}=(\frac{\Sigma \left(x_i-\mu \right)}{{\sigma }^2},\frac{\Sigma {\left(x_i-\mu \right)}^2}{2{\sigma }^4}-\frac{n}{2{\sigma }^2}).$

   Setting

   $\frac{\partial \ell }{\partial 𝜽}=\mathrm{𝟎}\text{ See score function}$

   and solve for $𝜽=(\mu ,{\sigma }^2)$ we have

   $\widehat{𝜽}=(\hat{\mu },{\hat{\sigma }}^2)=(\overline{x},\frac{n-1}{n}{s}^2),$

   where $\overline{x}=\Sigma x_i/n$ is the sample mean and $s^2=\Sigma {\left(x_i-\overline{x}\right)}^2/\left(n-1\right)$ is the sample variance. Finally, we verify that $\widehat{𝜽}$ is indeed the MLE of $𝜽$ by checking the negativity of the 2nd derivatives (for each parameter).