• ↑
  • STA260

STA260 Lecture 09

STA260 Lecture 09 Raw
STA260 Lecture 09 Flashcards

• Completed Notes Status

  • Completed insertions: 1
  • Ambiguities left unresolved: none

• Lecture Summary

  • Central objective: Apply the Maximum Likelihood Estimation (MLE) Principle to derive parameter estimators for common distributions and practice test preparation problems involving distributions and estimators.
  • Key concepts:
    • Maximum Likelihood Estimation (MLE) Principle: The procedure involves constructing the Likelihood Function, taking the logarithm to obtain the Log-Likelihood Function, differentiating with respect to parameters, and verifying the maximum condition using second derivatives.
    • Bernoulli Distribution MLE: For $X_1,\dots ,X_n\stackrel{iid}{\sim }\text{Bernoulli}(p)$, the Maximum Likelihood Estimator is ${\hat{p}}_{MLE}=\overline{X}$, derived by maximizing the Likelihood Function $L(p)=p^{\sum x_i}(1-p{)}^{n-\sum x_i}$.
    • Normal Distribution MLE: For $X_1,\dots ,X_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$, the MLE of the mean is ${\hat{\mu }}_{MLE}=\overline{X}$ and the MLE of the variance is ${\hat{\sigma }}_{MLE}^2=\frac{1}{n}\sum (X_i-\overline{X}{)}^2$ (the unadjusted sample variance).
    • Continuous Uniform Distribution MLE: For $X_1,\dots ,X_n\stackrel{iid}{\sim }\text{Uniform}(0,\theta )$, the MLE ${\hat{\theta }}_{MLE}=max\{X_1,\dots ,X_n\}$ is found by recognizing that the Likelihood Function is maximized at the smallest $\theta$ for which all observations lie within the support.
    • Test preparation: Problems involving Moment Generating Function to find distributions of sums, Central Limit Theorem applications, Chi-Square Distribution probability calculations, and verifying Unbiased Estimator properties and Consistent Estimator conditions for Method of Moments Estimator.
  • Connections:
    • The Bernoulli Distribution is equivalent to Binomial Distribution with $n=1$.
    • Moment Generating Function properties allow determination of the distribution of sums of independent random variables.
    • Central Limit Theorem justifies using normal approximations when sample size $n\ge 25$, while Chebyshev's Inequality provides bounds when the CLT cannot be applied.
    • The relationship $(n-1)S^2/{\sigma }^2\sim {\chi }_{n-1}^2$ connects sample variance to the Chi-Square Distribution.
    • Method of Moments Estimator for Gamma Distribution uses ${\mu }_1^{\prime }=\alpha \beta$ to derive estimators and their properties.

• Practice Questions

  • Remember/Understand:
    • What are the four steps in the Maximum Likelihood Estimation (MLE) Principle procedure?
    • What is the relationship between Bernoulli Distribution and Binomial Distribution?
    • State the condition under which $(n-1)S^2/{\sigma }^2$ follows a Chi-Square Distribution.
  • Apply/Analyze:
    • Given $X_1,\dots ,X_n\stackrel{iid}{\sim }\text{Poisson}(\lambda )$, use the Moment Generating Function to find the distribution of $T=\sum X_i$.
    • For $X_1,\dots ,X_{10}\stackrel{iid}{\sim }N(50,16)$, calculate $P(\overline{X}-50<2)$.
    • Verify whether ${\hat{\beta }}_{MOM}=\frac{\overline{X}}{2}$ is an Unbiased Estimator for $\beta$ when $X\sim \mathrm{\Gamma }(\alpha =2,\beta )$.
  • Evaluate/Create:
    • Explain why the Maximum Likelihood Estimator for $\theta$ in $\text{Uniform}(0,\theta )$ is $max\{X_1,\dots ,X_n\}$ rather than a solution obtained by setting the derivative of the Log-Likelihood Function to zero.
    • Compare the conditions required to apply the Central Limit Theorem versus Chebyshev's Inequality for a non-normal sample of size $n=9$.

• Challenging Concepts

  • Continuous Uniform Distribution MLE:
    • Why it's challenging: The Likelihood Function is non-differentiable at the boundary, and the MLE is found by examining the support constraints rather than solving $\frac{\partial \ell }{\partial \theta }=0$.
    • Study strategy: Sketch the Likelihood Function as a function of $\theta$ and visualize how it equals zero when $\theta <max\{X_i\}$ and is positive but decreasing for $\theta \ge max\{X_i\}$; recognize that boundary solutions arise when optimization constraints are active.
  • Normal Distribution variance MLE derivation:
    • Why it's challenging: The partial derivative with respect to ${\sigma }^2$ involves applying the chain rule to $(2\pi {\sigma }^2{)}^{-n/2}$ and $(2{\sigma }^2{)}^{-1}$ terms, requiring careful algebraic manipulation.
    • Study strategy: Practice differentiating functions of the form $(g(\theta ){)}^k$ and $1/g(\theta )$ repeatedly; verify each step by substituting the final MLE back into the first-order condition to confirm it equals zero.
  • Consistent Estimator verification:
    • Why it's challenging: Demonstrating consistency requires showing both that the estimator is Unbiased Estimator and that its variance vanishes as $n\to \mathrm{\infty }$, which involves understanding asymptotic properties.
    • Study strategy: Review Theorem 5 on sufficient conditions for consistency; practice computing $\underset{n\to \mathrm{\infty }}{lim}\text{Var}(\hat{\theta })$ for various estimators and linking unbiasedness to consistency.

• Action Plan

  • Immediate review actions:
    • Review the [COMPLETED] insertion in the Normal Distribution variance MLE section and verify against textbook or reference materials.
    • Re-read Lecture Summary and confirm all atomic note links are accurate.
    • Cross-reference the MLE derivations for Bernoulli Distribution, Normal Distribution, and Continuous Uniform Distribution with course materials.
  • Practice and application:
    • Answer Remember/Understand questions: recite the four-step MLE procedure, state the Bernoulli-Binomial relationship, and recall the chi-square condition.
    • Answer Apply/Analyze questions: work through the Poisson sum MGF derivation, compute the normal probability, and verify the unbiasedness of the gamma MOM estimator.
    • Attempt Evaluate/Create questions: write a detailed explanation of why the uniform MLE is at the boundary, and compare CLT versus Chebyshev's Inequality applicability.
  • Deep dive study:
    • Focus on the Continuous Uniform Distribution MLE and sketch the likelihood function for different values of $\theta$.
    • Rework the Normal Distribution variance MLE derivation step-by-step, verifying each derivative.
    • Extend the Moment Generating Function note with additional examples from this lecture.
    • Extend the Chi-Square Distribution note with the relationship to sample variance.
  • Verification and integration:
    • Compare the MLE and MOM approaches for the same distribution (e.g., Normal Distribution) to understand their differences.
    • Link this lecture to STA260 Lecture 08 (previous MLE introduction) and anticipate connections to STA260 Lecture 10 Raw.
    • Verify all test preparation problem solutions using R or by hand to ensure correctness.

• Footnotes

  • https://www.statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood