1 of 54

Density Estimation and �Gaussian Mixture Models

Lecture 5

Introducing Maximum Likelihood Estimation, and Gaussian Mixture Models

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

2 of 54

Join at slido.com�#1924089

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1924089

3 of 54

In The Last Lecture We…

  • Introduced k-means clustering as an unsupervised clustering algorithm.
    • Wanted a better way to capture uncertainty�in cluster predictions.
  • Reviewed basic ideas in probability.
  • Explored Bayesian updating to �determine the false-positive rate �needed for a wake-word detector.�

Listening!

Should these points be red or blue�or both?

1924089

4 of 54

Roadmap

  • Expectation and Variance
  • Density Estimation
    • Basic Probability Distributions
    • Maximum Likelihood
    • Bias
  • Multivariate Normal
  • Gaussian Mixture Models

Questions

1924089

5 of 54

Expectation and Variance

  • Expectation and Variance
  • Density Estimation
    • Basic Probability Distributions
    • Maximum Likelihood
    • Bias
  • Multivariate Normal
  • Gaussian Mixture Models

Questions

1924089

6 of 54

Expectations

  •  

1924089

7 of 54

Functions of Random Variables

  •  

1924089

8 of 54

Linearity of Expectation

  •  

 

 

 

 

 

1924089

9 of 54

Variance

  •  

 

 

 

 

 

1924089

10 of 54

Covariance

  •  

1924089

11 of 54

Probability Density Functions

  •  

1924089

12 of 54

Density Estimation

  • Expectation and Variance
  • Density Estimation
    • Basic Probability Distributions
    • Maximum Likelihood
    • Bias
  • Multivariate Normal
  • Gaussian Mixture Models

Questions

1924089

13 of 54

Density Estimation

  •  

1924089

14 of 54

Estimating the Parameters of a Distribution

  •  

1924089

15 of 54

The Likelihood Function

  •  

Independent

Identically

1924089

16 of 54

The Log Likelihood Function

  •  

1924089

17 of 54

Maximum Likelihood Estimation

  •  

1924089

18 of 54

Modeling: Bernoulli Distribution

  •  

 

 

 

 

 

 

 

 

 

1924089

19 of 54

The MLE for IID Bernoulli Samples (Part 1)

  •  

1924089

20 of 54

The MLE for IID Bernoulli Samples (Part 2)

  •  

Left as an exercise �(for your favorite AI).

1924089

21 of 54

The MLE for IID Bernoulli Samples (Part 3)

  •  

1924089

22 of 54

How can we mathematically formulate the problem of next-token prediction as a Maximum Likelihood Estimation (MLE) problem?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1924089

23 of 54

The MLE for ChatGPT

  •  

1924089

24 of 54

Normal (Gaussian) Distribution

  •  

 

1924089

25 of 54

The MLE for IID Gaussian Samples (Part 1)

  •  

1924089

26 of 54

The MLE for IID Gaussian Samples (Part 2)

  •  

1924089

27 of 54

The MLE for IID Gaussian Samples (Part 2)

  •  

 

 

1924089

28 of 54

Recap of the MLE for IID Gaussian Samples

  •  

Sufficient Statistics

1924089

29 of 54

Bias

  • Expectation and Variance
  • Density Estimation
    • Basic Probability Distributions
    • Maximum Likelihood
    • Bias
  • Multivariate Normal
  • Gaussian Mixture Models

Questions

1924089

30 of 54

Bias of an Estimate

  •  

1924089

31 of 54

Bias of the MLE for the Gaussian Distribution

  •  

Unbiased

 

Variance Identity

 

 

 

1924089

32 of 54

Bias of the MLE for the Gaussian Distribution

  •  

Biased

Unbiased

 

Sample Mean Variance

 

Variance Identity

1924089

33 of 54

How do we derive the formula for an unbiased estimator of the Gaussian variance?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

1924089

34 of 54

Multivariate Normal

  • Expectation and Variance
  • Density Estimation
    • Basic Probability Distributions
    • Maximum Likelihood
    • Bias
  • Multivariate Normal
  • Gaussian Mixture Models

Questions

1924089

35 of 54

Multivariate Gaussian Distribution

  •  

1924089

36 of 54

Demo

Gaussians

1924089

37 of 54

Gaussian Mixture Models

  • Expectation and Variance
  • Density Estimation
    • Basic Probability Distributions
    • Maximum Likelihood
    • Bias
  • Multivariate Normal
  • Gaussian Mixture Models

Questions

1924089

38 of 54

Gaussian Mixture Model (GMM)

  •  

 

 

 

1924089

39 of 54

Gaussian Mixture Model (GMM)

  •  

 

 

 

1924089

40 of 54

Demo

Gaussian Mixture Model

1924089

41 of 54

The GMM is a Latent Variable Model

  •  

1924089

42 of 54

The GMM is a Generative Model

  •  

 

 

1924089

43 of 54

Demo

Sampling from a GMM

1924089

44 of 54

Latent Variable Posteriors

  •  

Should these points be red or blue�or both?

1924089

45 of 54

Estimating the GMM Parameters

  •  

1924089

46 of 54

Quick Recap

If we knew the model parameters, we could easily compute the cluster assignments.

​

​

​

​

​

If we knew the cluster assignments, we could easily estimate the model parameters.

How can we solve this cyclic dependency?

 

Model Parameters

 

Cluster Assignments

1924089

47 of 54

The EM Algorithm

  •  

Easy to optimize joint probability

Current distribution �over the latent Z

Updates distribution over Z.

1924089

48 of 54

The EM Algorithm: E-step

  •  

 

N

K

1924089

49 of 54

 

  •  

 

Lagrangian for the �normalization constraint

 

 

1924089

50 of 54

 

  •  

1924089

51 of 54

 

  •  

1924089

52 of 54

The EM Algorithm for GMMs

  •  

1924089

53 of 54

Demo

Implement GMMs

1924089

54 of 54

Density Estimation and �Gaussian Mixture Models

Lecture 5

Credit: Joseph E. Gonzalez and Narges Norouzi

Reference Book Chapters:

  • Probability: Chapter 2.[1-2], 2.3.[1-3] (we will return to 2.4 onward later)
  • Distributions: Chapter 3.1, 3.2.1 (the rest of 3 is more advanced Gaussians)
  • Clustering: Chapter 15.1 (k-means), 15.2 (Gaussian Mixture Models)