STA260 Lecture 17

• Rao-Blackwell Theorem from STA260 Lecture 16
• Let $\hat{\theta }$ be an unbiased estimator. Finite variance.
• $T$ is a Sufficient Statistic for $\theta$
• Define ${\hat{\theta }}^{\ast }=\mathrm{E}[\hat{\theta }|T]$
  • $\hat{\theta }$ is an unbiased estimator
  • Proof that $\mathrm{Var}({\hat{\theta }}^{\ast })\le \mathrm{Var}(\hat{\theta })$
    • $\mathrm{Var}(\hat{\theta })=\mathrm{Var}(\mathrm{E}[\hat{\theta }|T])+\mathrm{E}[\mathrm{Var}(\hat{\theta }|T)]$
    • $\mathrm{Var}(\hat{\theta })-\mathrm{E}[\mathrm{Var}(\hat{\theta }|T)]=\mathrm{Var}(\mathrm{E}[\hat{\theta }|T])$
    • $\mathrm{Var}(\hat{\theta })-\mathrm{E}[\mathrm{Var}(\hat{\theta }|T)]=\mathrm{Var}({\hat{\theta }}^{\ast })$
    • Since $\mathrm{Var}(\hat{\theta }|T)\ge 0$
    • Since $\mathrm{E}[X]\ge 0$
    • It means that $\mathrm{Var}(\hat{\theta })\ge \mathrm{Var}({\hat{\theta }}^{\ast })$
      • Because we have to subtract a positive value from $\mathrm{Var}(\hat{\theta })$ to be equal to $\mathrm{Var}({\hat{\theta }}^{\ast })$
    • $\mathrm{Var}(\hat{\theta })\ge \mathrm{Var}({\hat{\theta }}^{\ast })$
• Minimal Sufficiency:
  • If $T(x)$ is sufficient for $f(T(x);\theta )$
  • If $Y(x)$ is sufficient for $f(Y(x);\theta )$
  • $\frac{f(T(x);\theta )}{f(Y(x);\theta )}$ is independent of $\theta$ iff $T(x)=Y(x)$
  • Then $T(x)$ is minimal sufficient for $\theta$
  • Intuition:
    • You can't really reduce your Sufficient Statistic any further without losing information about $\theta$
• MVUE - Minimum Variance Unbiased Estimator
  • If $T(x)$ is a sufficient statistic for $\theta$
  • If it have the minimum variance out of other unbiased estimators
    • You can apply the Rao-Blackwell Theorem to any unbiased estimator to get a new estimator with variance less than or equal to the original estimator
  • Then $\hat{\theta }$ is the MVUE for $\theta$
  • Example:
    • Let $X_1,\dots ,X_n$ be a random sample from $\mathrm{Exp}(\beta )$
    • Find the MVUE of $\mathrm{Var}(X_i)$
    • $f_X(x;\beta )={\beta }^{-1}\exp (-\frac{x}{\beta })$
    • Mean and var as $\beta ,{\beta }^2$
    • MVUE of $\mathrm{Var}(X_i)$ is the MVUE of ${\beta }^2$
    • 1: Find minimal sufficient statistic for $\beta$
      • For a sample
      • $L(\beta )=\prod _{i=1}^n{f}_{X_i}(x_i;\beta )$
      • $L(\beta )=\prod _{i=1}^n{\beta }^{-1}\exp (-\frac{x}{\beta })$
      • $L(\beta )={\beta }^{-n}\exp (-{\beta }^{-1}\sum _{i=1}^n{x}_i)$
      • $T(x)=\sum _{i=1}^n{X}_i$
      • $g(T(x)=\sum _{i=1}^n{x}_i,\theta =\beta )$
      • $h(x)=1$
      • So by Fisher's Factorization Theorem, $T(x)=\sum _{i=1}^n{X}_i$ is a sufficient statistic for $\beta$
      • We also have another stat that's valid
      • $L(\beta )={\beta }^{-n}\exp (-{\beta }^{-1}n\frac{\sum _{i=1}^n{x}_i}{n})$
      • $L(\beta )={\beta }^{-n}\exp (-{\beta }^{-1}n\overline{x})$
      • $g(T(x)=\overline{x},\theta =\beta )$
    • MVUE of ${\beta }^2$?
      • #tk show that $\mathrm{E}[\overline{X}]=\beta$
      • So let's consider $\mathrm{E}[{\overline{X}}^2]$ to see if we can get ${\beta }^2$
      • $\mathrm{Var}(\overline{X})=\mathrm{E}[{\overline{X}}^2]+\mathrm{E}[\overline{X}{]}^2$
      • $\mathrm{E}[{\overline{X}}^2]=\mathrm{Var}(\overline{X})+\mathrm{E}[\overline{X}{]}^2$
      • $\mathrm{Var}(\overline{X})+\mathrm{E}[\overline{X}{]}^2$
      • Since $\overline{X}\sim N({\mu }_{\overline{X}}=\beta ,{\sigma }_{\overline{X}}^2=\frac{{\beta }^2}{n})$
      • $\frac{{\sigma }^2}{n}+{\mu }^2$
      • $\frac{{\beta }^2}{n}+{\beta }^2$
      • $\frac{{\beta }^2}{n}+\frac{n{\beta }^2}{n}$
      • $\frac{n{\beta }^2+{\beta }^2}{n}$
      • $\frac{{\beta }^2(n+1)}{n}$
      • $\frac{n+1}{n}{\beta }^2$
      • We don't get ${\beta }^2$, but we get $\frac{n+1}{n}{\beta }^2$
      • $\mathrm{E}[{\overline{X}}^2]=\frac{n+1}{n}{\beta }^2$
      • $n\mathrm{E}[{\overline{X}}^2]=(n+1){\beta }^2$
      • $\frac{n}{n+1}\mathrm{E}[{\overline{X}}^2]={\beta }^2$
      • $\mathrm{E}[\frac{n}{n+1}{\overline{X}}^2]={\beta }^2$
      • This is unbiased.
      • We have an unbiased estimator for ${\beta }^2$
      • Since $\mathrm{E}[g(T)|T]=g(t)$
      • We have a function of our Sufficient Statistic.
      • $\frac{n{\overline{x}}^2}{n+1}$ is unbiased for ${\beta }^2$
      • How do we show that it's the MVUE?
    • Rao-Blackwell
      • We can apply the Rao-Blackwell Theorem to any unbiased estimator to get a new estimator with variance less than or equal to the original estimator
      • $\hat{\theta }=\frac{n}{n+1}{\overline{X}}^2$
      • $T(x)=\overline{x}$
      • ${\hat{\theta }}^{\ast }=\mathrm{E}[\hat{\theta }|T]$
      • ${\hat{\theta }}^{\ast }=\mathrm{E}[\frac{n}{n+1}{\overline{X}}^2|\overline{x}]=\frac{n{\overline{x}}^2}{n+1}$
      • So by Rao-Blackwell it's the MVUE.
  • Example:
    • Bernoulli
    • $X_1,\dots ,X_n\sim \mathrm{Bernoulli}(p)$
    • Find the MVUE of $p(1-p)$
    • Mean and variance is $p$ and $p(1-p)$
    • So we need something unbiased for $p(1-p)$
    • By Fisher's Factorization Theorem we can find a sufficient statistic for $p$.
    • $L(p)=\prod _{i=1}^n{p}^{x_i}(1-p{)}^{1-x_i}$
    • $L(p)=p^{\sum _{i=1}^n{x}_i}(1-p{)}^{n-\sum _{i=1}^n{x}_i}$
    • $L(p)=p^{\sum _{i=1}^n{x}_i}(1-p{)}^n(1-p{)}^{-\sum _{i=1}^n{x}_i}$
    • $L(p)=p^{\sum _{i=1}^n{x}_i}(1-p{)}^n{\left(\frac{1}{1-p}\right)}^{\sum _{i=1}^n{x}_i}$
    • $L(p)={\left(\frac{p}{1-p}\right)}^{\sum _{i=1}^n{x}_i}(1-p{)}^n$
    • $g(T(x)=\sum _{i=1}^n{x}_i,\theta =p)={\left(\frac{p}{1-p}\right)}^{\sum _{i=1}^n{x}_i}(1-p{)}^n$
    • $h(x)=1$
    • Since $X_i\sim \mathrm{Bernoulli}(p)$
    • Then sum is $\mathrm{Bin}(n,p)$
      • Aside: If $Y_i\sim \mathrm{Exp}(\beta )$ then $\sum _{i=1}^n{Y}_i\sim \mathrm{\Gamma }(\alpha =1,\beta )$
    • Unbiased Estimator for $p(1-p)$
    • $x=0⟹f(x)=1-p$
    • $x=1⟹f(x)=p$
    • $f(X_1=1,X_2=0)=f(X_1=1)f(X_2=0)$
    • $=p(1-p)$
    • This Joint Distribution is an unbiased estimator for $p(1-p)$
    • $W=\{\begin{array}{ll}1& x_1=1,x_2=0\\ 0& \text{otherwise}\end{array}$
    • $\mathrm{E}[W]=\sum _{i=1}^{n}xf(x)=1\cdot f(W=1)=p(1-p)$
    • Since $W$ isn't a function of the Sufficient Statistic
    • Also $T(x)=\sum _{i=1}^n{x}_i=n\overline{x}=x_1+x_2+\sum _{i=3}^n{x}_i$
    • We can apply the Rao-Blackwell Theorem to get a new estimator with variance less than or equal to the original Estimator
    • We can then find the conditional expectation of the Unbiased Estimator $W$ given a Sufficient Statistic $T(x)$
    • $\mathrm{E}[W|T=t]$
    • $\mathrm{E}[X_1=1,X_2=0|T=t]$
    • $\frac{P(X_1=1,X_2=0,T=t)}{P(T=t)}$
    • First two terms:
      • $T(x)=\sum _{i=1}^n{x}_i$ number of successes
      • $1+0+\cdots =t$
      • $1+(t-1)=t$
      • $x_1+x_2+\cdots +x_n=t$
    • $\frac{P(X_1=1)P(X_2=0)P(\sum _{i=3}^n{x}_i=t-1)}{P(T=t)}$
    • $\frac{p(1-p)P(\sum _{i=3}^n{x}_i=t-1)}{P(T=t)}$
    • $P\left(\sum _{i=2}^n{x}_i\right)$ is binomial with $n-2$ and $x=t-1$
    • $\frac{p(1-p)(\genfrac{}{}{0ex}{}{n-2}{t-1})p^{t-1}(1-p{)}^{n-2-(t-1)}}{P(T=t)}$
    • $\frac{p(1-p)(\genfrac{}{}{0ex}{}{n-2}{t-1})p^{t-1}(1-p{)}^{n-t-1}}{P(T=t)}$
    • $\frac{(1-p)(\genfrac{}{}{0ex}{}{n-2}{t})p^t(1-p{)}^{n-t-1}}{P(T=t)}$
    • $\frac{(\genfrac{}{}{0ex}{}{n-2}{t})p^t(1-p{)}^{n-t}}{P(T=t)}$
    • $\frac{(\genfrac{}{}{0ex}{}{n-2}{t})p^t(1-p{)}^{n-t}}{(\genfrac{}{}{0ex}{}{n}{t})p^t(1-p{)}^{n-t}}$
    • $\frac{(\genfrac{}{}{0ex}{}{n-2}{t})}{(\genfrac{}{}{0ex}{}{n}{t})}$
    • $\frac{\frac{(n-2)!}{t!(n-2-t)!}}{\frac{n!}{t!(n-t)!}}=\frac{(-n+t)(-n+t+1)}{n(n-1)}$
    • $=\frac{t(n-t)}{n(n-1)}$
    • $=\frac{n\overline{x}(n-n\overline{x})}{n(n-1)}$
    • $=\frac{n\overline{x}(n(1-\overline{x}))}{n(n-1)}$
    • $=\frac{n\overline{x}(1-\overline{x})}{n-1}$
    • By Rao-Blackwell, $\frac{n\overline{x}(1-\overline{x})}{n-1}$ is the MVUE for $p(1-p)$
• Review:
  • Consider power law dist:
    • $f_X(x;\alpha )=(\alpha -1)x_{min}^{\alpha -1}{x}^{-\alpha }x\ge x_{min},\alpha >3$
  • a: for $x_{min}=1$ find the MLE for $\alpha$
    • $f_X(x;\alpha )=(\alpha -1)x^{-\alpha }$ for $x\ge 1,\alpha >3$
    • $L(\alpha )=\prod _{i=1}^n(\alpha -1)x^{-\alpha }$
    • $L(\alpha )=(\alpha -1{)}^n\prod _{i=1}^n{x}_i^{-\alpha }$
    • $\ell (\alpha )=\log L(\alpha )$
    • $\ell (\alpha )=n\log (\alpha -1)-\sum _{i=1}^n\alpha \log x_i$
    • $\ell (\alpha )=n\log (\alpha -1)-\alpha \sum _{i=1}^n\log x_i$
    • $\frac{\partial \ell (\alpha )}{\partial \alpha }=n(\alpha -1{)}^{-1}-\sum _{i=1}^n\log x_i$
    • $n(\alpha -1{)}^{-1}=\sum _{i=1}^n\log x_i$
    • $\frac{n}{\sum _{i=1}^n\log x_i}=\alpha -1$
    • ${\hat{\alpha }}_{MLE}=1+\frac{n}{\sum _{i=1}^n\log x_i}$
  • b: Find Fisher Information
    • Assume for a point
    • $f_X(x;\alpha )=(\alpha -1)x^{-\alpha }$
    • $\ell (\alpha )=\log f_X(x;\alpha )$
    • $\ell (\alpha )=\log (\alpha -1)-\alpha \log x$
    • $\frac{\partial \ell (\alpha )}{\partial \alpha }=\frac{1}{\alpha -1}-\log x$
    • $\frac{\partial \ell (\alpha )}{\partial \alpha }=(\alpha -1{)}^{-1}-\log x$
    • $\frac{{\partial }^2\ell (\alpha )}{\partial {\alpha }^2}=-(\alpha -1{)}^{-2}$
    • $I(\alpha )=-\mathrm{E}[-(\alpha -1{)}^{-2}]$
    • $I(\alpha )=\mathrm{E}[(\alpha -1{)}^{-2}]=(\alpha -1{)}^{-2}$
    • $I(\alpha )=(\alpha -1{)}^{-2}$
  • c: Find the CRLB for sample of size $n$
    • $CRLB=\frac{1}{nI(\alpha )}$
    • $\frac{(\alpha -1{)}^2}{n}$
    • $\mathrm{Var}(\hat{\alpha })\ge \frac{(\alpha -1{)}^2}{n}$
• 2
  • Let $X_1,\dots ,X_n\stackrel{iid}{\sim }\mathrm{\Gamma }(\alpha =2,\beta )$
  • $f_X(x_i;\beta )=\frac{x}{{\beta }^2}\exp (-\frac{x}{\beta })$
  • a: Find a minimal Sufficient Statistic for $\beta$
    • Fisher's Factorization Theorem
    • $L(\beta )=\prod _{i=1}^{n}x{\beta }^{-2}\exp (-x{\beta }^{-1})$
    • $L(\beta )={\beta }^{-2n}\exp (-{\beta }^{-1}\sum _{i=1}^n{x}_i)\prod _{i=1}^n{x}_i$
    • $L(\beta )={\beta }^{-2n}\exp \left({\beta }^{-1}\sum _{i=1}^n{x}_i\right)\prod _{i=1}^n{x}_i$
    • #tk why is it ${\beta }^{-1}$ and not $-{\beta }^{-1}$
    • $g(T(x),\beta )$
      • $T(x)=\sum _{i=1}^n{x}_i$
    • $h(x)=\prod _{i=1}^n{x}_i$
    • Now show minimal sufficiency
    • $\frac{L(\beta ;X_i)}{L(\beta ;Y_i)}=\frac{{\beta }^{-2n}\exp (-{\beta }^{-1}\sum _{i=1}^n{x}_i)\prod _{i=1}^n{x}_i}{{\beta }^{-2n}\exp (-{\beta }^{-1}\sum _{i=1}^n{y}_i)\prod _{i=1}^n{y}_i}$
    • $=\frac{\prod _{i=1}^n{x}_i}{\prod _{i=1}^n{y}_i}\exp \left({\beta }^{-1}(\sum _{i=1}^n{x}_i-\sum _{i=1}^n{y}_i)\right)$
    • If $\sum _{i=1}^n{x}_i-\sum _{i=1}^n{y}_i=0$
    • Then $\exp (0)=1$
    • To be independent of $\beta$ we need that $\sum _{i=1}^n{x}_i-\sum _{i=1}^n{y}_i=0$
    • So $\sum _{i=1}^n{x}_i=\sum _{i=1}^n{y}_i$
    • So $T(x)=T(y)$
    • So it's minimal Sufficient Statistic
  • b: Show gamma dist above is a member of generalized exponential family
    • $f_X(x;\theta )=\exp (-c(\theta )T(x)+d(\theta )+S(x))$
    • PDF of gamma is $f_X(x;\alpha ,\beta )=\frac{x}{{\beta }^2}\exp (-\frac{x}{\beta })$
    • $f_X(x;\alpha ,\beta )=\frac{x}{{\beta }^2}\exp (-\frac{x}{\beta })$
    • $f_X(x;\alpha ,\beta )=\exp \ln \left(\frac{x}{{\beta }^2}\right)\exp (-\frac{x}{\beta })$
    • $f_X(x;\alpha ,\beta )=\exp (\ln x-2\ln \beta )\exp (-\frac{x}{\beta })$
    • $f_X(x;\alpha ,\beta )=\exp (-\frac{x}{\beta }-2\ln \beta +\ln x)$
    • $f_X(x;\alpha ,\beta )=\exp (-\frac{1}{\beta }x-2\ln \beta +\ln x)$
    • $-c(\beta )=-\frac{1}{\beta }$
    • $T(x)=x$
    • $d(\beta )=-2\ln \beta$
    • $S(x)=\ln x$
  • c: Find the MVUE using Rao-Blackwell for $\beta$
    • Sufficient Statistic $T(x)=\sum _{i=1}^n=x_i$
      • Since $X_i\sim \mathrm{\Gamma }(\alpha =2,\beta )$
      • $\mathrm{E}[X_i]=2\beta$
    • Unbiased Estimator for $\beta$.
    • $\mathrm{E}[T(x)]=\mathrm{E}\left[\sum _{i=1}^n{x}_i\right]=\mathrm{E}[n\overline{x}]=\sum _{i=1}^n\mathrm{E}[x_i]=\sum _{i=1}^{n}2\beta =2\beta n$
    • This is an unbiased estimator for $2\beta n$
    • $\mathrm{E}[T(x)]=2n\beta$
    • $\frac{1}{2n}\mathrm{E}[T(x)]=\beta$
    • $\mathrm{E}[\frac{1}{2n}T(x)]=\beta$
    • $\mathrm{E}[\frac{1}{2n}n\overline{x}]=\beta$
    • $\mathrm{E}[\frac{1}{2}\overline{x}]=\beta$
    • This is an unbiased estimator for $\beta$
    • To use Rao-Blackwell, we can use $\mathrm{E}[g(T)|T]=g(T)$
    • $\mathrm{E}[\frac{1}{2}\overline{x}|T]=\frac{1}{2}\overline{x}$
    • $T$ is sufficient for $\beta$ and $\frac{1}{2}\overline{x}$ is an unbiased estimator for $\beta$. The $\mathrm{E}[\frac{1}{2}\overline{x}|T]=\frac{1}{2}\overline{x}$
    • By Rao-Blackwell, $\frac{1}{2}\overline{x}$ is the MVUE for $\beta$.
    • ${\hat{\beta }}_{MVUE}=\frac{1}{2}\overline{x}=\frac{1}{2n}T$