# Why cannot MLE be implemented for Gaussian mixture model directly?

Consider the following density, the mixture of two Gaussian distributions, \begin{align*} p(x)= p(k=1) N(x|\mu_1,\sigma^2_1) + p(k=0) N(x|\mu_0,\sigma^2_0) , \end{align*} where $$p(k=1)+p(k=0)=\pi_1+\pi_0=1$$ and $$N(x|\mu,\sigma^2)$$ is the density of Gaussian distribution with mean $$\mu$$ and variance $$\sigma^2$$. Parameters of interests are $$\pi_0$$, $$\mu_i$$'s and $$\sigma^2_i$$'s.

This Q & A shows the MLE for the mixture of two Gaussian distributions when the latent variables $$K_i$$'s are observed. In this question, suppose we only observe $$X_i$$'s, and the latent variables $$K_i$$'s are unobserved. Classical methods for estimation of these $$5$$ unknown parameters are EM-algorithm and MCMC sampling, see Hastie et. al. (2009) for details.

Why cannot MLE be implemented for Gaussian mixture model directly?

(Some attempt)

The log-likelihood would be \begin{align*} \ln P(x|\theta) = \sum_{i=1}^n \bigg[ (1-k_i) (\ln \pi_0 + \ln N(x_i|\mu_0,\sigma_0^2))+k_i(\ln \pi_1 + \ln N(x_i|\mu_1,\sigma_1^2)) \bigg]. \end{align*}

• There is no closed form solution for the MLE. This means that the loglikelihood needs to be maximised numerically. This is what EM in fact does. One can use alternative numerical approaches, but they are not necessarily better, and don't necessarily deserve the description "directly" either.. Mar 25, 2022 at 15:48
• For simple problems, maximizing the full log-likelihood you give here numerically works fine. (So the short answer to "Why cannot ..." is "It can.") Mar 25, 2022 at 16:16
• The real problem is that the likelihood blows up whenever one of the $\mu_i$ coincides with a data value and the corresponding $\sigma_i$ shrinks to zero. This gives multiple global maxima, all of which are usually considered unrealistic.
– whuber
Mar 25, 2022 at 16:48
• I agree with Ben on this. It is not the end of the world. Especially with a reasonably large sample and some regularisation, it is doable. Mar 26, 2022 at 1:21
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 13, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.9169193506240845, "perplexity": 890.8053285298421}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2023-06/segments/1674764501555.34/warc/CC-MAIN-20230209081052-20230209111052-00688.warc.gz"}