Parameter Estimation

Fall 2023

Instructor:

Ajit Rajwade

​

Topic Overview

• Concept of maximum likelihood estimation
• Maximum likelihood estimates of the parameters of various distributions
• Concept of point estimate and confidence intervals
• Estimator bias, variance and mean-squared error

​

​

​

What is parameter estimation?

• In many applications, we have samples of a random variable which we know belongs to a distribution of so and so family (eg: Poisson, normal, etc.)

​

• However, we may not know some or any of the parameters of the distribution.

​

• We can however estimate the parameters given some assumptions about the samples. This is called parameter estimation.

Parameter estimation

• Let X be a continuous random variable with probability density f_X(x;θ).

​

• Consider a sample of X having value say x₁. Then we say that the likelihood of this sample is f_X (x₁;θ) assuming parameter θ.

​

• If we observe some n samples (rather n samples, one from each of the n iid random variables having density f_X(x;θ)), then their joint likelihood is given as

f _{X1,X2,X3,…,Xn}(x₁, x₂,…, x_n;θ) or in

shorthand f (x₁, x₂,…, x_n;θ).

Parameter estimation

• Now consider the case where we have the n sample values but we do not know θ.

​

• A reasonable way to estimate θ is to treat the joint likelihood as a function of θ and maximize it with respect to θ.

​

• Intuitively, such an estimate is the one which maximizes the probability density of the observed values. It is called as the maximum likelihood estimate of θ.

Parameter estimation

• Maximum likelihood estimation is equally applicable for discrete random variables in which case the likelihood is defined in terms of discrete probabilities instead of probability density functions.

​

• In maximum likelihood estimation, it is often easier (in terms of calculations) to maximize log f (x₁, x₂,…, x_n;θ)

instead of f (x₁, x₂,…, x_n;θ).

​

• Doing so does not alter the estimate in any way for many densities that are not zero-valued except at +/- infinity.

Parameter estimation

• But how does one get a functional form for the joint likelihood f (x₁, x₂,…, x_n;θ)?

​

• One typically assumes that the samples are statistically independent.

​

• This is a reasonable assumption we shall use, though there are situations where it is violated (we will not see those situations in this course).

Examples (Derivations on the board and in the book)

• Maximum likelihood estimation of p parameter of a Bernoulli distribution
• Of the mean/variance λ of a Poisson distribution
• Of the mean and standard deviation of a Gaussian distribution
• Of the range [α,θ] of a uniform distribution.
• ML with a twist: linear regression.

​

All derivations scanned and uploaded on moodle (also in the lecture videos).

ML estimates are random variables!

• The ML estimate is a random variable!

​

• Why? Because its value is a function of the samples from some underlying distribution.

​

• The ML estimate has its own probability density function, and its own mean, variance, etc.

​

• When you compute the value of an ML estimate, it is the value of a sample of that random variable.

Confidence intervals

• The value of the ML estimate of a parameter is a single scalar which may differ from the true parameter value. Such an estimate is called a point estimate.

​

• We do expect/wish the point estimate to be close to the true value – especially when the number of samples is high.

​

• One way to see this closeness, is to construct an interval around the ML estimate and show that the true value will lie inside this interval with high probability. This is called a confidence interval.

True parameter

Estimated parameter

Standard deviation of the estimated parameter

Confidence interval: empirical mean of a Gaussian (known variance)

• You know the following fact:

​

​

​

• Hence

Two-sided 99% confidence interval:

Known σ

Empirical mean of Gaussian samples

See clarification on next two slides

Clarification

• For Gaussian samples, the empirical mean given by

is always Gaussian distributed. Why?

​

• If X and Y are independent random variables, then Z = X+Y has a pdf given by:

​

​

• If X and Y are Gaussian distributed, then it can be shown that Z is also Gaussian distributed (easy using MGFs, a bit tedious using basic integral calculus within the above formula).

​

This is called a convolution operation. It is widely used in image/signal processing for other reasons. It is commutative, i.e. you can swap f_X and f_Y here.

Clarification

• This result extends to the sum of more than two Gaussian random variables as well, i.e. the sum(and hence average) of n > 2 Gaussian random variables is also a Gaussian random variable.

​

• Note: the empirical mean of any n i.i.d. random variables has an approximate (for finite n) Gaussian distribution by Central Limit Theorem.

​

• But if the original n random variables are themselves Gaussian, the empirical mean exactly has a Gaussian distribution.

Confidence interval: empirical mean of a Gaussian (known variance)

• You know that the following fact:

​

​

• Hence

(two-sided) 99% confidence interval:

Known σ

Empirical mean of Gaussian samples

Note that this analysis and hence this confidence interval is not applicable in the case when the σ is unknown and hence needs to be estimated. In fact, the following random variable does not have a normal distribution but a student-t distribution instead – which we have not covered in class so far:

Confidence interval: empirical mean of a Gaussian (known variance)

• Sometimes we may be interested in whether the true parameter exceeds its estimate with q% confidence.

​

• Hence

(upper one-sided) 99% confidence interval

Why is this 2.35 instead of 2.5?

Confidence interval: a clarification

• With 99% probability, our estimate (which is a random variable) will be such that the true parameter happens to lie inside a confidence interval around it (the estimated value).

Confidence interval: variance of a Gaussian

• Given samples from a normal distribution with unknown parameters, we know that

(approximate) Confidence interval: Mean of a Bernoulli Random variable

• Let X be the number of successes in a sequence of n Bernoulli trials with success probability p.

​

• X is a binomial random variable, and can be approximated as a normal random variable with mean np and variance np(1-p). Hence we have:

(approximate) Confidence interval: Mean of a Bernoulli Random variable

(approximate) Confidence interval: Mean of a Bernoulli Random variable

Estimator bias, variance and mean squared error

• Let X= (X₁, X₂, …, X_n) be a set of n i.i.d. random variables from a given distribution with parameter θ.

​

• Suppose θ’ is an estimate of θ (obtained through some estimator).

​

• Note that the ML estimator is one particular estimator of θ, but there exist many other types of estimators.

​

• How does one conclude whether or not a given estimator is a good estimator?

Estimator bias, variance and mean squared error

• One metric for this is to evaluate (θ’-θ)².

​

• But this is a random variable! So we can consider its expected value, i.e. E[(θ’-θ)²].

​

• This is called the mean squared error of the estimator.

​

• We desire estimators with low mean-squared error.

Estimator bias, variance and mean squared error

• In some estimators, the expected value of the estimate, i.e. E(θ’) may not be equal to the true parameter θ. Such estimators are called biased.

​

• The following quantity is called as the bias of the estimator: E(θ’) – θ.

​

• Example of unbiased estimator (bias = 0): ML estimator for the mean of a Gaussian, Bernoulli, etc.

​

• Example of biased estimator: ML estimator of the variance of a Gaussian when mean is unknown, interval of a uniform distribution.

Estimator bias, variance and mean squared error

• Merely knowing the estimator bias does not give us a complete idea of the quality of the estimator.

​

• What if different samples of X gave wildly different estimates of θ for the same estimator?

​

• Such estimators are said to have a high variance.

​

• More formally, the variance of an estimator is defined as : E[(θ’-E(θ’)²].

Estimator bias, variance and mean squared error

• It can be proved that (proof on board and in book):

​

​

​

​

​

• Point to note: a biased estimator may have lower MSE than an unbiased estimator (because the latter may have higher variance).

variance

Squared bias

Estimator bias, variance and mean squared error

• Examples of estimator bias, variance and MSE – on the board!

​

• Consider n samples X₁, X₂, ..., X_n from some distribution with parameter θ.

​

• An estimator of the form θ’ = 1 has very high bias (unless of course the true value of θ was 1) but very low variance (why?).

​

• Suppose θ represented the expected value of the samples. Then an estimator of the form θ’’ = X_i will have no bias (why?) but variance equal to that of X_i. In particular note, that the variance of θ’’ does not decrease with the number of samples n – which is not desirable.

Estimator consistency

• Let θ be the parameter of a distribution. Let an estimator of this parameter produce value .

​

• We say that the estimator is (asymptotically) consistent if

​

​

• The estimates θ’ and θ’’ on the previous slide are not consistent estimators.

​

• Note an estimator may be biased but still consistent (eg: ML estimator for variance of a Gaussian with unknown mean). And an unbiased estimator may be inconsistent (eg: θ’’).

Motivation for MLE

• The MLE is a consistent estimator as long as the parameter values being estimated do not vary with n.

​

• No consistent estimators can achieve a lower asymptotic MSE than the MLE.

​

• We state these properties without proof.