Maximum Likelihood Estimation – Python Guide – Analytics India Magazine

Data is in all places. The current human life-style depends closely on information. Machine studying is a big area that strives onerous repeatedly to make nice issues out of the largely out there information. With information in hand, a machine studying algorithm tries to seek out the sample or the distribution of that information. Machine studying algorithms are often outlined and derived in a pattern-specific or a distribution-specific method. For occasion, Logistic Regression is a conventional machine studying algorithm meant particularly for a binary classification downside. Linear Regression is a conventional machine studying algorithm meant for the info that’s linearly distributed in a multi-dimensional area. One particular algorithm can’t be utilized for an issue of various nature.

To this finish, Maximum Likelihood Estimation, merely often known as MLE, is a conventional probabilistic method that may be utilized to information belonging to any distribution, i.e., Normal, Poisson, Bernoulli, and so forth. With prior assumption or data in regards to the information distribution, Maximum Likelihood Estimation helps discover probably the most likely-to-occur distribution parameters. For occasion, allow us to say we now have information that’s assumed to be usually distributed, however we have no idea its imply and commonplace deviation parameters. Maximum Likelihood Estimation iteratively searches the more than likely imply and commonplace deviation that would have generated the distribution. Moreover, Maximum Likelihood Estimation might be utilized to each regression and classification issues. 

Therefore, Maximum Likelihood Estimation is solely an optimization algorithm that searches for probably the most appropriate parameters. Since we all know the info distribution a priori, the algorithm makes an attempt iteratively to seek out its sample. The method is way generalized, in order that you will need to devise a user-defined Python perform that solves the actual machine studying downside. 

How does Maximum Likelihood Estimation work?

The time period chance might be outlined as the chance that the parameters into account might generate the info. A chance perform is solely the joint probability function of the info distribution. A most chance perform is the optimized chance perform employed with most-likely parameters. Function maximization is carried out by differentiating the chance perform with respect to the distribution parameters and set individually to zero.

If we glance again into the fundamentals of likelihood, we will perceive that the joint likelihood perform is solely the product of the likelihood features of particular person information factors. With a big dataset, it’s virtually troublesome to formulate a joint likelihood perform and differentiate it with respect to the parameters. Hence MLE introduces logarithmic chance features. Maximizing a strictly growing perform is similar as maximizing its logarithmic type. The parameters obtained by way of both chance perform or log-likelihood perform are the identical. The logarithmic type permits the massive product perform to be transformed right into a summation perform. It is kind of simple to sum the person chance features and differentiate it. Because of this simplicity in math works, Maximum Likelihood Estimation solves enormous datasets with information factors within the order of tens of millions!

For every downside, the customers are required to formulate the mannequin and distribution perform to reach on the log-likelihood perform. The optimization is carried out utilizing the SciPy library’s ‘optimize’ module. The module has a way referred to as ‘minimize’ that may reduce any enter perform with respect to an enter parameter. In our case, the MLE appears for maximizing the log-likelihood perform. Therefore, we provide the unfavorable log chance because the enter perform to the ‘minimize’ methodology. It differentiates the user-defined unfavorable log-likelihood perform with respect to every enter parameter and arrives on the optimum parameters iteratively. The parameters which might be discovered via the MLE method are referred to as most chance estimates.

In the sequel, we focus on the Python implementation of Maximum Likelihood Estimation with an instance. 

Regression on Normally Distributed Data

Here, we carry out easy linear regression on artificial information. The information is ensured to be usually distributed by incorporating some random Gaussian noises. Data might be stated to be usually distributed if its residual follows the traditional distribution—Import the required libraries.

 import numpy as np
 import pandas as pd
 from matplotlib import pyplot as plt
 import seaborn as sns
 from statsmodels import api
 from scipy import stats
 from scipy.optimize import reduce 

Generate some artificial information primarily based on the idea of Normal Distribution.

 # generate an unbiased variable 
 x = np.linspace(-10, 30, 100)
 # generate a usually distributed residual
 e = np.random.regular(10, 5, 100)
 # generate floor reality
 y = 10 + 4*x + e
 df = pd.DataBody('x':x, 'y':y)


Visualize the artificial information on Seaborn’s regression plot.

 sns.regplot(x='x', y='y', information = df)


Plot: Normal distribution

The information is often distributed, and the output variable is a repeatedly various quantity. Hence, we will use the Ordinary Least Squares (OLS) methodology to find out the mannequin parameters and use them as a benchmark to judge the Maximum Likelihood Estimation method. Apply the OLS algorithm to the artificial information and discover the mannequin parameters.

 options = api.add_constant(df.x)
 mannequin = api.OLS(y, options).match()


OLS results

We get the intercept and regression coefficient values of the straightforward linear regression mannequin. Further, we will derive the usual deviation of the traditional distribution with the next codes.

 res = mannequin.resid
 standard_dev = np.std(res)


See Also

LSTM Autoencoder

As we now have solved the straightforward linear regression downside with an OLS mannequin, it’s time to clear up the identical downside by formulating it with Maximum Likelihood Estimation.

Define a user-defined Python perform that may be iteratively referred to as to find out the unfavorable log-likelihood worth. The key thought of formulating this perform is that it should include two parts: the primary is the mannequin constructing equation (right here, the straightforward linear regression). The second is the logarithmic worth of the likelihood density perform (right here, the log PDF of regular distribution). Since we want unfavorable log-likelihood, it’s obtained simply by negating the log-likelihood.

 # MLE perform
 # ml modeling and neg LL calculation
 def MLE_Norm(parameters):
   # extract parameters
   const, beta, std_dev = parameters
   # predict the output
   pred = const + beta*x
   # Calculate the log-likelihood for regular distribution
   LL = np.sum(stats.norm.logpdf(y, pred, std_dev))
   # Calculate the unfavorable log-likelihood
   neg_LL = -1*LL
   return neg_LL 

Minimize the unfavorable log-likelihood of the generated information utilizing the reduce methodology out there with SciPy’s optimize module.

 # reduce arguments: perform, intial_guess_of_parameters, methodology
 mle_model = reduce(MLE_Norm, np.array([2,2,2]), methodology='L-BFGS-B')


Maximum Likelihood Estimation results

The MLE method arrives on the closing optimum answer after 35 iterations. The mannequin’s parameters, the intercept, the regression coefficient and the usual deviation are nicely matching to these obtained utilizing the OLS method. 

This Colab Notebook incorporates the above code implementation.

Here comes the large query. If the OLS method offers the identical outcomes with none tedious perform formulation, why will we go for the MLE method? The reply is that the OLS method is totally problem-specific and data-oriented. It cannot be used for a special type of downside or a special information distribution. On the opposite hand, the MLE method is a basic template for any type of downside. With experience in Maximum Likelihood Estimation, customers can formulate and clear up their very own machine studying issues with uncooked information in hand.

Wrapping up

In this tutorial, we mentioned the idea behind the Maximum Likelihood Estimation and the way it may be utilized to any type of machine studying downside with structural information. We mentioned the chance perform, log-likelihood perform, and unfavorable log-likelihood perform and its minimization to seek out the utmost chance estimates. We went via a hands-on Python implementation on fixing a linear regression downside that has usually distributed information. Users can do extra follow by fixing their machine studying issues with MLE formulation. 

Further studying on Maximum Likelihood Estimation

Subscribe to our Newsletter

Get the most recent updates and related provides by sharing your e mail.

Join Our Telegram Group. Be a part of a fascinating on-line group. Join Here.


Please enter your comment!
Please enter your name here