STA260 Lecture 13

• If we have multiple estimators with similar features. Which is the best to use?
  • Suppose we have $X_1,\dots ,X_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$
  • With $n\ge 2$
  • $\overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$
    • $\mathrm{E}[\overline{X}]=\mu$
      • Unbiased
    • $\mathrm{Var}(\overline{X})=\frac{{\sigma }^2}{n}$
  • $\tilde{X}=\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)+\frac{1}{n}=\overline{X}+\frac{1}{n}$
    • $\mathrm{E}[\tilde{X}]=\mu +\frac{1}{n}$
      • Biased
    • #tk practice this derivation
  • $\dot{X}=\frac{X_1+X_2}{2}$
    • $\mathrm{E}[\dot{X}]=\mu$
      • Unbiased
  • Which estimator is better?
  • You want your estimator to have small variance. Always.
• Efficiency
  • The efficiency of statistical estimators refers to how well they utilize data
  • The criteria used to determine efficiency is the variance of the estimator. An efficient estimator has the smallest variance among all unbiased estimators.
  • We may need to examine asymptotic or limiting behaviour of the estimator.
  • If we only have unbiased estimators:
    • You just need to compare the variances of the estimators. The one with the smallest variance is the most efficient.
    • Either directly or through ratios.
      • $\frac{\mathrm{Var}(X)}{\mathrm{Var}(Y)}<1$ means that $X$ is more efficient than $Y$.
      • $\frac{\mathrm{Var}(X)}{\mathrm{Var}(Y)}=1$ means that $X$ and $Y$ are equally efficient.
      • $\frac{\mathrm{Var}(X)}{\mathrm{Var}(Y)}>1$ means that $Y$ is more efficient than $X$.
  • If we have biased estimators:
    • We need to take into account the bias as well as the variances.
    • Trade-off with estimators when we have data.
    • MSE: Mean Squared Error
      • $\mathrm{MSE}[\hat{\theta }]=\mathrm{E}[(\hat{\theta }-\theta {)}^2]$
      • $\mathrm{MSE}(\hat{\theta })=\mathrm{Var}(\hat{\theta })+{(\mathrm{Bias}(\hat{\theta }))}^2$
      • We want to minimize the MSE.
      • You can either have a tight estimator but not centered around the true value, or you can have an estimator that is centered around the true value but has a large variance. You want to find a balance between these two.
• Example:
  • We have $X_1,X_2,X_3$ from $\mathrm{Exp}(\theta )$
  • $f_X(x;\theta )=\frac{1}{\theta }{e}^{-\frac{x}{\theta }}$ for $x>0$
  • ${\hat{\theta }}_1=X_1$
  • ${\hat{\theta }}_2=\frac{X_1+X_2}{2}$
  • ${\hat{\theta }}_3=\frac{X_1+2X_2}{3}$
  • ${\hat{\theta }}_4=min\{X_1,X_2,X_3\}$
  • ${\hat{\theta }}_5=\overline{X}$
  • Which are unbiased?:
  • We know $\mathrm{E}[X]=\theta$ and $\mathrm{Var}(X)={\theta }^2$
    • ${\hat{\theta }}_1=X_1$
      • $\mathrm{E}[{\hat{\theta }}_1]=\mathrm{E}[X_1]=\theta$
    • ${\hat{\theta }}_2=\frac{X_1+X_2}{2}$
      • $\mathrm{E}[{\hat{\theta }}_2]=\mathrm{E}\left[\frac{X_1+X_2}{2}\right]=\frac{1}{2}\mathrm{E}[X_1+X_2]=\frac{1}{2}(\mathrm{E}[X_1]+\mathrm{E}[X_2])$
      • $=\frac{1}{2}(\theta +\theta )=\theta$
    • ${\hat{\theta }}_3$
      • $\mathrm{E}\left[\frac{X_1+2X_2}{3}\right]=\frac{1}{3}\mathrm{E}[X_1+2X_2]=\frac{1}{3}(\mathrm{E}[X_1]+2\mathrm{E}[X_2])$
      • $=\frac{1}{3}(\theta +2\theta )=\theta$
    • ${\hat{\theta }}_4$
      • $\mathrm{E}[{\hat{\theta }}_4]=\mathrm{E}[min\{X_1,X_2,X_3\}]$
      • Without loss of generality, say $X_1$ is the minimum.
        • #tk ask in office hours if this is valid. Why or why not. Why do we use the CDF instead of MGF? Why was I wrong?
      • $\mathrm{E}[X_1]=\theta$
      • Use the CDF technique to find the distribution of the minimum.
      • First CDF of Exp:
        • $F_X(x)=P(X\le x)=1-e^{-\frac{x}{\theta }}$ for $x>0$
          • $={\int }_{-\mathrm{\infty }}^{x}f(t)\text{differential}t$
          • $={\int }_{-\mathrm{\infty }}^x\frac{1}{\theta }{e}^{-\frac{t}{\theta }}\text{differential}t$
          • $\dots$
        • $P(X\le x)=F_X(x)=1-e^{-\frac{x}{\theta }}$
        • $P(X>x)=1-F_X(x)=e^{-\frac{x}{\theta }}$
      • The CDF of the minimum:
        • ${\hat{\theta }}_4$
        • $F_{{\hat{\theta }}_4}(x)=P({\hat{\theta }}_4\le x)$
        • $=P(min\{X_1,X_2,X_3\}\le x)$
        • Take the complement.
        • $=1-P(min\{X_1,X_2,X_3\}>x)$
          • If $min\{X_1,X_2,X_3\}$ is larger than a value $x$. Then all data points are larger than $x$.
        • $=1-P(X_1>x,X_2>x,X_3>x)$
        • By independence (random sample):
        • $=1-P(X_1>x)P(X_2>x)P(X_3>x)$
        • We know $P(X>x)=e^{-\frac{x}{\theta }}$
        • $=1-{\left(e^{-\frac{x}{\theta }}\right)}^3$
        • $=1-e^{-\frac{3x}{\theta }}$
        • $\tilde{\theta }=\frac{{\hat{\theta }}_4}{3}$
        • $=1-e^{-\frac{x}{\tilde{\theta }}}$
      • So the CDF of the minimum is $F_{{\hat{\theta }}_4}=1-e^{-\frac{x}{\tilde{\theta }}}=1-e^{-\frac{3x}{\theta }}$
      • The PDF of the minimum the derivative of the CDF:
      • $f_{{\hat{\theta }}_4}(x)=\frac{\text{differential}}{\text{differential}x}{F}_{{\hat{\theta }}_4}(x)$
      • $f_{{\hat{\theta }}_4}(x)=\frac{\text{differential}}{\text{differential}x}(1-e^{-\frac{3x}{\theta }})$
      • $f_{{\hat{\theta }}_4}(x)=\frac{3}{\theta }{e}^{\frac{-3x}{\theta }}$
      • $\tilde{\theta }=\frac{\theta }{3}$
      • $f_{{\hat{\theta }}_4}(x)={\displaystyle \frac{1}{\tilde{\theta }}}{e}^{-\frac{x}{\tilde{\theta }}}\sim \mathrm{Exp}\left(\frac{\theta }{3}\right)$
      • $\mathrm{E}[{\hat{\theta }}_4]=\mathrm{E}[min\{X_1,X_2,X_3\}]=\mathrm{E}\left[\mathrm{Exp}\left(\frac{\theta }{3}\right)\right]=\frac{\theta }{3}$
      • Biased.
      • Finding how likely we are to find the smallest data rather than the smallest of the actual data.
    • ${\hat{\theta }}_5$
      • $\mathrm{E}[\overline{X}]=\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]$
      • We already know this is $\theta$.
  • From 4 unbiased estimators, which is the best? That with the least variance.
    • ${\hat{\theta }}_1=X_1$
      • $\mathrm{Var}(X_1)={\theta }^2$
    • ${\hat{\theta }}_2=\frac{X_1+X_2}{2}$
      • $\mathrm{Var}({\hat{\theta }}_2)=\mathrm{Var}\left(\frac{X_1+X_2}{2}\right)=\frac{1}{4}\mathrm{Var}(X_1+X_2)$
      • By independence:
      • $\frac{1}{4}(\mathrm{Var}(X_1)+\mathrm{Var}(X_2))$
      • $\frac{1}{4}({\theta }^2+{\theta }^2)$
      • $\frac{2{\theta }^2}{4}=\frac{{\theta }^2}{2}$
    • ${\hat{\theta }}_3=\frac{X_1+2X_2}{3}$
      • $\mathrm{Var}({\hat{\theta }}_3)=\mathrm{Var}\left(\frac{X_1+2X_2}{3}\right)=\frac{1}{9}\mathrm{Var}(X_1+2X_2)$
      • By independence:
      • $\frac{1}{9}(\mathrm{Var}(X_1)+4\mathrm{Var}(X_2))$
      • $\frac{1}{9}({\theta }^2+4{\theta }^2)$
      • $\frac{5{\theta }^2}{9}$
    • ${\hat{\theta }}_4=min\{X_1,X_2,X_3\}$
      • $\mathrm{Var}({\hat{\theta }}_4)=\mathrm{Var}(\mathrm{Exp}(\tilde{\theta }))=\mathrm{Var}\left(\mathrm{Exp}\left(\frac{\theta }{3}\right)\right)={\left(\frac{\theta }{3}\right)}^2$
      • $=\frac{{\theta }^2}{9}$
      • #tk is this the right derivation?
    • ${\hat{\theta }}_5=\overline{X}$
      • $\mathrm{Var}({\hat{\theta }}_5)=\mathrm{Var}(\overline{X})=\frac{{\theta }^2}{n}=\frac{{\theta }^2}{3}$
    • ${\hat{\theta }}_4$ is the most efficient estimator, since it has the smallest variance.
• Example 2:
  • $X_1,\dots ,X_n$ are random samples from $U(0,\theta )$
  • ${\hat{\theta }}_1=2\overline{X}$
  • ${\hat{\theta }}_2=\frac{n+1}{n}{X}_{(n)}$
    • $X_{(n)}=max\{X_1,\dots ,X_n\}$
  • Sidenote:
    • Familiar with:
      • We have $X_1,\dots ,X_n$ as data points with no other information.
    • Ordered Statistics:
      • $X_{(1)},X_{(2)},\dots ,X_{(n)}$
      • This isn't the first sample point, this is the smallest sampled value.
      • So $X_{(1)}\le X_{(2)}\le \cdots \le X_{(n)}$
      • $X_{(n)}$ is the largest sampled value.
  • Show both are unbiased.
    • ${\hat{\theta }}_1$
      • $\mathrm{E}[{\hat{\theta }}_1]=\mathrm{E}[2\overline{X}]=2\mathrm{E}[\overline{X}]=\frac{2\theta }{2}=\theta$
    • ${\hat{\theta }}_2=\frac{n+1}{n}{X}_{(n)}$
      • $\mathrm{E}\left[\frac{n+1}{n}{X}_{(n)}\right]$
      • We don' t know $n$
      • So we need to find the distribution of $\frac{n+1}{n}{X}_{(n)}$
      • PDF of Uniform:
        • $f_X(x)=\frac{1}{\theta }$ for $0<x<\theta$
        • Mean: $\frac{\theta }{2}$
        • Variance: $\frac{{\theta }^2}{12}$
      • $\frac{n+1}{n}{X}_{(n)}$
      • If $max\{X_1,\dots ,X_n\}<x$ then $x$ is greater than all the data points.
      • CDF:
        • $F_{X_{(n)}}(x)=P(X_{(n)}\le x)$
        • $F_{X_{(n)}}(x)=P(max\{X_1,\dots ,X_n\}\le x)$
        • $F_{X_{(n)}}(x)=P(X_1<x,X_2<x<\dots ,X_n<x)$
        • By independence:
        • $F_{X_{(n)}}(x)=P(X_1<x)P(X_2<x)\dots P(X_n<x)$
        • Single CDF:
          • $P(X<x)=F_X(x)=\frac{x}{\theta }$ for $0<x<\theta$
        • $F_{X_{(n)}}(x)={\left(\frac{x}{\theta }\right)}^n$
        • $F_{X_{(n)}}(x)=\frac{x^n}{{\theta }^n}$
      • PDF:
        • $f_{X_{(n)}}(x)=\frac{\text{differential}}{\text{differential}x}{F}_{X_{(n)}}(x)$
        • $f_{X_{(n)}}(x)=\frac{\text{differential}}{\text{differential}x}\left(\frac{x^n}{{\theta }^n}\right)$
        • $f_{X_{(n)}}(x)=\frac{n{x}^{n-1}}{{\theta }^n}$
        • $f_{X_{(n)}}(x)={\theta }^{-n}n{x}^{n-1}$
      • $\mathrm{E}[X_{(n)}]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x{f}_X(x;\theta )\text{differential}x$
      • $\mathrm{E}[X_{(n)}]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\theta }^{-n}n{x}^n\text{differential}x$
      • $\mathrm{E}[X_{(n)}]=n{\theta }^{-n}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{x}^n\text{differential}x$
      • $\mathrm{E}[X_{(n)}]=n{\theta }^{-n}\left[\frac{1}{n+1}{x}^{n+1}\right]{}_{-\mathrm{\infty }}^{\mathrm{\infty }}$
      • $\dots$
      • $=\frac{n\theta }{n+1}$
      • $\frac{n+1}{n}\mathrm{E}[X_{(n)}]$
      • $\frac{n+1}{n}\frac{n\theta }{n+1}$
      • $\theta$
      • So it's unbiased.
    • Another way to have found expectation:
      • $f_{X_{(n)}}(x)=\frac{n{x}^{n-1}}{{\theta }^n}$
      • Try to manipulate this to become a beta dist.
      • Since we have $0<x<\theta$ we need $0<x<1$
      • $=\frac{n{\left(\frac{x}{\theta }\right)}^{n-1}}{{\theta }^n}$ for $0<x<1$
      • $=\frac{n}{{\theta }^n}\frac{x^{n-1}}{{\theta }^{-(n-1)}}$
      • $=\frac{n}{\theta }{x}^{n-1}(1-x{)}^{1-1}$
      • $\beta =1$
      • $\alpha =n$
      • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=\frac{\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(n)\mathrm{\Gamma }(1)}$
      • For any integer, $\mathrm{\Gamma }(n+1)=n\mathrm{\Gamma }(n)$ or $\mathrm{\Gamma }(n)=(n-1)!$
      • #tk maybe wrong?
        • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=\frac{(n+1-1)!}{(n-1)!0!}$
        • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=\frac{n!}{(n-1)!}$
        • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=\frac{n(n-1)!}{(n-1)!}$
        • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=n$
      • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=\frac{n\mathrm{\Gamma }(n)}{\mathrm{\Gamma }(n)}$
      • $\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )+\mathrm{\Gamma }(\beta )}=n$
      • $=\frac{1}{\theta }\frac{\mathrm{\Gamma }(n-1)}{\mathrm{\Gamma }(n)\mathrm{\Gamma }(1)}{x}^{n-1}(1-x{)}^{1-1}$ for $0<x<1$
      • $\beta =1$
      • $\alpha =n$
      • Scaled by ${\theta }^{-1}$
      • $X_{(n)}\sim \theta \cdot \mathrm{Beta}(n,1)$
      • for a beta:
        • Mean: $\frac{\alpha }{\alpha +\beta }$
        • Variance: $\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}$
      • $\frac{n+1}{n}\mathrm{E}[\theta X_{(n)}]$
      • $\frac{n+1}{n}\theta \frac{n}{n+1}$
      • $\theta$
    • ${\sigma }^2=\mathrm{Var}(\theta X_{(n)})={\theta }^2\mathrm{Var}(X_{(n)})$
      • ${\theta }^2\frac{n}{(n+1{)}^2(n+2)}$