The Maximum Likelihood Principle in Machine Learning

• Bayes Theorem Explained
• Bayes Rule in Machine Learning

I suggest reading them before tackling the next ones to follow along with the beautiful story-line that we are creating together. Once you are ready, sit back, relax, and lets learn all about what is MLE and how it is used in Machine Learning.

The Maximum Likelihood principle

MLE estimates parameters within a chosen model family. It does not establish that the family is correct or automatically make predictions generalize to new data; those claims require separate checks.

As a simplified classification illustration, suppose we fit two class-conditional height densities. Comparing their density values alone corresponds to a decision rule with equal class priors and equal misclassification costs. It is not the same as estimating posterior class probabilities.

In the illustration, the density for the female-labeled class is higher at 172 cm. Under equal priors and equal error costs, that rule chooses this class. Height alone cannot reliably determine a person’s sex or gender.

That’s very cool you might say, but how do we actually calculate these probability distributions? Do not worry, we will get to it right now. First we will explain the general process behind the Maximum Likelihood principle, and then we will go through a more specific case with an example.

Related concepts: compare the likelihood with the posterior in our Bayes rule guide. Write down which quantities are observed and which are being estimated before doing the calculation.

Calculating the distributions: estimating a parametric density function

As usual in Machine Learning, the first thing we need to start calculating a distribution is something to learn from: our precious data. We will denote our data vector of size n, as X. In this vector each of the rows is a data point with d features, therefore our data vector X is actually a vector of vectors: a matrix of size n x d; n data points with d features each.

Once we have collected the data which we want to calculate a distribution from, we need to start guessing. Guessing? Yep, you read right. We need to guess the kind of density function or distribution which we think our data follows: Gaussian, Exponential, Poisson…

Don’t worry tough, this might not sound very scientific, but most times for every kind of data there is a distribution which is most likely to fit best: Gaussian for features like temperature or height, exponential for features regarding time, like length of phone calls or the life of bacterial populations, or Poisson (great video on The Poisson Distribution here) for features like the number of houses sold in a specific period of time.

Once this is done we calculate the specific parameters of the chosen distribution that best fit our data. For a normal distribution this would be the mean and the variance. As the gaussian or normal distribution is probably the easiest one to explain and understand, we will continue this post assuming we have chosen a gaussian density function to represent our data.

If you don’t know what a Gaussian distribution is, a gentle book like Bayesian Statistics the Fun Way is a great place to start. You can find our review of the book here: Bayesian Statistics the Fun Way review.

Alright, lets continue with the Maximum Likelihood principle and our example:

For a multivariate normal distribution with d features, estimate d means and d(d + 1)/2 distinct covariance-matrix entries. These include d variances on the diagonal and d(d − 1)/2 covariances off the diagonal.

Write the parameter set as θ, including the mean vector and covariance matrix in the multivariate normal case. The likelihood L(θ; X) evaluates the joint mass or density of the observed data X as a function of θ. A density value is not the probability of an exact continuous observation.

In this notation X is the data matrix, and X(1) up to X(n) are each of the data points, and θ is the given parameter set for the distribution. Again, as the goal of the Maximum Likelihood Principle is to chose the parameter values so that the observed data is as likely as possible, we arrive at an optimisation problem dependent on θ.

To obtain this optimal parameter set, we take derivatives with respect to θ in the likelihood function and search for the maximum: this maximum represents the values of the parameters that make observing the available data as likely as possible.

Now, if the data points of X are independent of each other, the likelihood function can be expressed as the product of the individual probabilities of each data point given the parameter set:

Taking the derivatives with respect to this equation for each parameter (mean, variance,etc…) keeping the others constant, gives us the relationship between the value of the data points, the number of data points, and each parameter.

Lets look at an example of how this is done using the normal distribution, and an easy male height dataset.

A deeper look into the maths of the Maximum Likelihood Principle using a normal distribution

Lets see an example of how to use Maximum Likelihood to fit a normal distribution to a set of data points with only one feature: height in centimetres. As we mentioned earlier, there are to parameters that we have to calculate: the mean and the variance.

For this, we have to know the density function for the normal distribution:

Once we know this, we can calculate the likelihood function for each data point. For the first data point it would be:

For the whole data set, considering our data points as independent and we can therefore calculate the likelihood function as the product of the likelihoods of the individual points, it would be:

We can take this function and express it in a logarithmic way, which facilitates posterior calculations and yields exactly the same results.

Finally, we set the derivative of the likelihood function with regards to the mean to zero, reaching an expression where we obtain the value of this first parameter:

Surprise! The maximum likelihood estimate for the mean of the normal distribution is just what we would intuitively expect: the sum of the value of each data point divided by the number of data points.

Now that we have calculated the estimate for the mean, it is time to do the same for the other relevant parameter: the variance. For this, just like before, we take derivatives in the likelihood function with the goal of finding the value of the variance that maximises the likelihood of the observed data.

The normal-model MLE for variance divides the sum of squared deviations by n. The familiar unbiased sample-variance estimator divides by n − 1 instead. These are different estimators, even though they become close for large samples.

That is it! We have seen the general mathematics and procedures behind the calculation Maximum Likelihood estimate of a normal distribution. Lets look at a quick numeric example to finish off!

Maximum Likelihood estimate for male heights: a numeric example

Lets the very simple example we have mentioned earlier: we have a data set of male heights in a certain area, and we want to find an optimal distribution to it using Maximum Likelihood.

If we remember right, the first step for this (after collecting and understanding the data) is to choose the shape of the density function that we want to estimate. In our case, for height, we will use a Gaussian distribution, which we also saw in the general reasoning behind the maths of ML. Lets retake a look at the formula that defines such distribution:

Also, lets recover the likelihood function for just one point of the data set.

Imagine our data vector X, in our case is the following:

We have 10 data points (n = 10) and one feature for each data point (d=1). If in the formula shown above for each of the data points we substitute their actual values we get something like:

For example, evaluate the normal density at heights of 176 cm and 172 cm using a mean of 180 cm and a variance of 4 cm² (standard deviation 2 cm). Variance uses squared units; density values are not probabilities of those exact heights.

After this quick note, if we continue with the procedure to obtain the maximum likelihood estimate that best fits out data set, we would have to first calculate the mean. For our case it is very simple: we just sum up the values of the data points and divide this sum by the number of data points.

If we do the same for the variance, calculating the squared sum of the value of each data point minus the mean and dividing it by the total number of points we get:

With the fitted mean and variance, we can evaluate the density at a new height such as 177 cm. To obtain a probability, integrate that density over a height interval; a density at one point is not a probability of dataset membership.

Now, if we had another data set, with female heights for example, and we did the same procedure, we would have two height distributions: one for male and one for females.

For classification using two fitted class-conditional densities, combine each density with its class prior and normalize to obtain posterior class probabilities. Comparing densities alone assumes equal priors; unequal decision costs can also change the appropriate decision rule.

Summary: The Maximum Likelihood Principle

We have seen what Maximum Likelihood is, the maths behind it, and how it can be applied to solve real world problems. This has given us the basis to tackle the next post, that you have all been asking for: The maths behind Bayes’ Theorem, which are very similar to Maximum Likelihood.

Read it here: Bayes Theorem Calculation: The Maths of Machine Learning. And to get new posts as they come out, follow us on X!

Additional Resources

In case you want to go more in depth into Maximum Likelihood and Machine Learning, check out these other resources:

• Video with a very clear explanation and example of Maximum Likelihood
• Very detailed video about Maximum Likelihood for the normal distribution
• Maximum Likelihood slides
• A list of the best Probability Online Courses out there.
• Great Data Analysis and probability book reviews:
  • Probability for the enthusiastic beginner
  • Statistical Rethinking
  • Think Stats

As always, contact us at howtolearnmachinelearning@gmail.com if you have any questions. Have a fantastic day and keep learning!

Also, don’t forget to check out our awesome Machine Learning videos!

Tags: Probability, Probability and Statistics, Probability and Machine Learning, Maximum Likelihood Principle , What is MLE, What is Maximum Likelihood Estimation.