STA260 Lecture 16

• MVUE
• Blackwell
• Exponential Family
  • Family of dists
  • $f_X(x;\theta )=\{\begin{array}{ll}{e}^{-c(\theta )T(x)+d(\theta )+S(x)}& \text{for all }x\in \mathcal{S}\\ 0& \text{Otherwise}\end{array}$
  • The natural / canonical form is slightly different.
    • If you want to show something is in the form of an exponential family, you can show it in the natural form or the above.
  • Here $\theta$ can mean multiple parameters.
  • General Exponential Family:
    • A distribution function is a member of the general exponential family if it can be expressed in the form as follows.
    • $e^x=\exp (x)$
    • $f_X(x;\theta )=\exp (-c(\theta )T(x)+d(\theta )+S(x))$
    • $c(\theta )$ is a function of parameters.
    • $T(x)$ is a function of the data, it's a Sufficient Statistic
    • $d(\theta )$ is a function of parameters.
    • $S(x)$ is a function of the data.
    • Note:
      • $T(x)=1,T(x)=0$ are not valid sufficient statistics, because it doesn't capture any information about the data.
      • Sufficient Statistic
      • $x=\exp (\ln (x))$ is a useful trick.
    • Example:
      • Show Exponential distribution with parameter $\beta$ is a member of the GEF (General Exponential Family).
      • $f_X(x;\theta )=\exp (-c(\theta )T(x)+d(\theta )+S(x))$
        • #tk remember this for test.
      • $f_X(x;\beta )=\frac{1}{\beta }{e}^{-\frac{x}{\beta }}$
      • $f_X(x;\beta )=\frac{1}{\beta }\exp (-\frac{x}{\beta })$
      • $f_X(x;\beta )=\frac{1}{\beta }\exp (-\frac{1}{\beta }\cdot x)$
      • $f_X(x;\beta )=\exp (\ln \left(\frac{1}{\beta }\right))\exp (-\frac{1}{\beta }\cdot x)$
      • $f_X(x;\beta )=\exp (-\frac{1}{\beta }\cdot x+\ln \left(\frac{1}{\beta }\right)+0)$
      • $T(x)=x$ is non-trivial.
      • $-c(\beta )=-{\beta }^{-1}$
      • $d(\beta )=\ln \left(\frac{1}{\beta }\right)$
      • $S(x)=0$
        • We didn't have a restriction stating it must be non-trivial, so it's fine.
      • So it is a member of the GEF.
    • Example:
      • Binomial
      • Show that the Binomial distribution with parameters $p$ is a member of the GEF.
      • $f_X(x;\theta )=\exp (-c(\theta )T(x)+d(\theta )+S(x))$
      • It's only by parameter $p$
      • $f_X(x;p)=(\genfrac{}{}{0ex}{}{n}{x})p^x(1-p{)}^{n-x}$
      • $f_X(x;p)=\exp (\ln ((\genfrac{}{}{0ex}{}{n}{x})p^x(1-p{)}^{n-x}))$
      • $f_X(x;p)=\exp (\ln (\genfrac{}{}{0ex}{}{n}{x})+x\ln p+(n-x)\ln (1-p))$
      • $n\ln (1-p)-x\ln (1-p)$
      • $f_X(x;p)=\exp (\ln (\genfrac{}{}{0ex}{}{n}{x})+x\ln p+n\ln (1-p)-x\ln (1-p))$
      • $f_X(x;p)=\exp (\ln (\genfrac{}{}{0ex}{}{n}{x})+x\ln p-x\ln (1-p)+n\ln (1-p))$
      • $f_X(x;p)=\exp (\ln (\genfrac{}{}{0ex}{}{n}{x})+x(\ln p-\ln (1-p))+n\ln (1-p))$
      • $T(x)=x$
      • $c(p)=\ln p-\ln (1-p)$
      • $-c(p)=\ln (1-p)-\ln p$
      • $T(x)=-x$
      • $f_X(x;p)=\exp (\ln (\genfrac{}{}{0ex}{}{n}{x})-x(\ln (1-p)-\ln p)+n\ln (1-p))$
      • $f_X(x;p)=\exp (\ln (\genfrac{}{}{0ex}{}{n}{x})-x\ln \frac{1-p}{p}+n\ln (1-p))$
      • $d(p)=n\ln (1-p)$
      • $S(x)=\ln (\genfrac{}{}{0ex}{}{n}{x})$
      • So it is a member of the GEF.
      • #tk check if this is accurate.
    • Example:
      • Normal distribution with known variance ${\sigma }^2$ and unknown mean $\mu$.
      • Show that it is a member of the GEF.
      • $f_X(x;\theta )=\exp (-c(\theta )T(x)+d(\theta )+S(x))$
      • $f_X(x;\mu )=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$
      • $f_X(x;\mu )=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$
      • $f_X(x;\mu )=\exp (\ln \frac{1}{\sqrt{2\pi {\sigma }^2}})\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$
      • $f_X(x;\mu )=\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2}-\frac{1}{2}\ln \left(2\pi {\sigma }^2\right))$
      • $f_X(x;\mu )=\exp (-\frac{(x-\mu )(x-\mu )}{2{\sigma }^2}-\frac{1}{2}\ln \left(2\pi {\sigma }^2\right))$
      • $f_X(x;\mu )=\exp (-\frac{(x-\mu )(x-\mu )}{2{\sigma }^2}-\frac{1}{2}\ln \left(2\pi {\sigma }^2\right))$
      • #tk
      • $-\frac{(x-\mu )(x-\mu )}{2{\sigma }^2}=-\frac{x^2-2\mu x+{\mu }^2}{2{\sigma }^2}$
      • $-\frac{(x-\mu )(x-\mu )}{2{\sigma }^2}=-\frac{x^2}{2{\sigma }^2}-\frac{\mu x}{{\sigma }^2}+\frac{{\mu }^2}{2{\sigma }^2}$
      • $f_X(x;\mu )=\exp (-\frac{x^2}{2{\sigma }^2}-\frac{\mu x}{{\sigma }^2}+\frac{{\mu }^2}{2{\sigma }^2}-\frac{1}{2}\ln \left(2\pi {\sigma }^2\right))$
      • $f_X(x;\mu )=\exp (-\frac{x^2}{2{\sigma }^2}-x\frac{\mu }{{\sigma }^2}+\frac{{\mu }^2}{2{\sigma }^2}-\frac{1}{2}\ln \left(2\pi {\sigma }^2\right))$
      • $T(x)=x$
      • $-c(\mu )=-\frac{\mu }{{\sigma }^2}$
      • $d(\mu )=\frac{{\mu }^2}{2{\sigma }^2}$
      • $S(x)=-\frac{x^2}{2{\sigma }^2}-\frac{1}{2}\ln (2\pi {\sigma }^2)$
      • #tk check if this is it.
    • Example:
      • Poisson
      • Show that the Poisson distribution with parameter $\lambda$ is a member of the GEF.
      • $f_X(x;\theta )=\exp (-c(\theta )T(x)+d(\theta )+S(x))$
      • $f_X(x;\lambda )=\frac{e^{-\lambda }{\lambda }^x}{x!}$
      • $f_X(x;\lambda )=\frac{\exp (-\lambda ){\lambda }^x}{x!}$
      • $f_X(x;\lambda )=\frac{\exp (-\lambda )\exp (\ln ({\lambda }^x))}{\exp (\ln (x!))}$
      • $f_X(x;\lambda )=\frac{\exp (-\lambda )\exp (x\ln (\lambda ))}{\exp (\ln x!)}$
      • $f_X(x;\lambda )=\frac{\exp (-\lambda +x\ln \lambda )}{\exp (\ln x!)}$
      • $f_X(x;\lambda )=\exp (-\lambda +x\ln \lambda -\ln x!)$
      • $f_X(x;\lambda )=\exp (\ln \lambda x-\lambda -\ln x!)$
      • $f_X(x;\lambda )=\exp ((-\ln \lambda )(-x)-\lambda -\ln x!)$
      • $T(x)=-x$
      • $-c(\lambda )=-\ln \lambda$
      • $d(\lambda )=-\lambda$
      • $S(x)=-\ln x!$
    • Why is uniform not GEF?
      • There's nothing to get a non-trivial function of the data, since there is no data.
      • $f_X(x;\theta )=\frac{1}{\theta }$ for $0,\theta$
      • $\exp (\ln ({\theta }^{-1}))$
      • $\exp (-\ln (\theta )\cdot 1+0+0)$
      • Since uniform involves the parameter and data, then we can't have the Sufficient Statistic be a non-trivial function of the data, so it can't be a member of the GEF.
• Minimum Variance Unbiased Estimator (MVUE)
  • An unbiased estimator $\hat{\theta }$ is the MVUE for $\theta$ if $\mathrm{Var}(\hat{\theta })\le \mathrm{Var}(\tilde{\theta })$ for all unbiased estimators $\tilde{\theta }$ of $\theta$.
  • So it's the best unbiased estimator, in terms of variance.
  • Cramér-Rao Lower Bound
  • Let $X_1,\dots ,X_n$ be a random sample from a common dist with parameter $\theta$.
    • $X_1,\dots ,X_n\stackrel{iid}{\sim }{f}_X(x;\theta )$
  • An estimator $\hat{g}(X_1,\dots ,X_n)$ is an MVUE for parameter $\theta$ if:
    • $\hat{g}$ is unbiased for $\theta$.
      • $\mathrm{E}[\hat{g}(X_1,\dots ,X_n)]=g(\theta )$
    • For any other estimator $\hat{h}(X_1,\dots ,X_n)$ the $\mathrm{Var}(\hat{g}(X_1,\dots ,X_n))\le \mathrm{Var}(\hat{h}(X_1,\dots ,X_n))$
      • The most efficient unbiased estimator.
      • $\hat{g}$ has the minimum variance.
  • Definition is in the name.
  • Found using the Rao-Blackwell Theorem.
  • Properties:
    • For any random variable $T$ and function $g$. The expected value $\mathrm{E}[g(T)|T]=g(T)$.
• Proof of Rao-Blackwell Theorem:
  • Let $\hat{\theta }$ be an unbiased estimator of $\theta$ where $\mathrm{Var}(\hat{\theta })<\mathrm{\infty }$. If $T$ is a Sufficient Statistic for $\theta$, ${\hat{\theta }}^{\ast }=\mathrm{E}[\hat{\theta }|T]$, then.
  • $\mathrm{E}[{\hat{\theta }}^{\ast }]=\theta$
  • $\mathrm{Var}({\hat{\theta }}^{\ast })\le \mathrm{Var}(\hat{\theta })$
  • Review from STA256
    • Let $X,Y$ be random variables in a sample space.
      • Law of total expectation: $\mathrm{E}[X]=\mathrm{E}[\mathrm{E}[X|Y]]$
        • #tk remember this for finals.
      • Law of total variance: $\mathrm{Var}(X)=\mathrm{Var}(\mathrm{E}[X|Y])+\mathrm{E}[\mathrm{Var}(X|Y)]$
  • Proof:
    • $\mathrm{E}[{\hat{\theta }}^{\ast }]=\mathrm{E}[\mathrm{E}[\hat{\theta }|T]]$
    • By law of total expectation, $\mathrm{E}[{\hat{\theta }}^{\ast }]=\mathrm{E}[\hat{\theta }]$
    • $\mathrm{E}[{\hat{\theta }}^{\ast }]=\mathrm{E}[\hat{\theta }]$
    • Since it's unbiased
    • $\mathrm{E}[{\hat{\theta }}^{\ast }]=\theta$
    • $\mathrm{Var}(\hat{\theta })=\mathrm{Var}(\mathrm{E}[\hat{\theta }|T])+\mathrm{E}[\mathrm{Var}(\hat{\theta }|T)]$
    • $\mathrm{Var}(\hat{\theta })\ge \mathrm{Var}(\mathrm{E}[\hat{\theta }|T])$
    • $\mathrm{Var}(\hat{\theta })\ge \mathrm{Var}({\hat{\theta }}^{\ast })$