STA258 Lecture 12

Hypothesis Testing
- Example:
  - You're measuring lifetime of mice under different medications.
  - You can hypothesize that $μ = 72$ hours of lifetime.
  - This is the population mean.
  - Use a random sample to test the validity of the hypothesis.
  - Say you have $Y_{1}, \dots, Y_{10}$
  - We can use the $\bar{Y}$
  - Say we get $\bar{Y} = 73$
    - We don't have sufficient evidence to reject the hypothesis.
  - Suppose $\bar{Y} = 500$
    - We have sufficient evidence to reject the hypothesis.
  - Suppose $\bar{Y} = 1000$
    - We have sufficient evidence to reject the hypothesis.
  - But with $\bar{Y} = 500$ we have less evidence to reject, than with $\bar{Y} = 1000$ .
  - Decision rule, if $\bar{Y} > 100$ we reject the hypothesis that $μ = 72$ .
  - If $min {Y_{1}, \dots, Y_{n}} > 100$ , we reject the hypothesis.
    - This one is stronger than the sample mean one.
- Example:
  - $Y_{1}, \dots, Y_{n} \overset{i i d}{\sim} f_{Y} (y; θ) θ \in Ω$
  - Where $Ω$ is the total parameter space.
  - Goal is to test the hypothesis that $H_{0} : θ \in Ω_{0}$ vs $H_{a} : θ \in Ω_{a}$
  - $H_{0}$ is our null hypothesis, and $H_{a}$ is our alternative hypothesis.
    - CSC207 Assignment 1
    - $Ω_{0} \cup Ω_{a} = Ω$
- Rat Example:
  - Same as above
  - $H_{0} : μ = 72$
  - $H_{a} = μ > 72$
  - $H_{0} : θ = 72$
    - Simple hypothesis, point hypothesis, singleton set.
  - $H_{a} : θ > 72$
    - Composite hypothesis, interval hypothesis, not a point hypothesis, not a singleton set.
  - $D = space {Y_{1}, \dots, Y_{n}}$ is the space of sample.
  - Decision rule.
  - A test of $H_{0}$ vs $H_{a}$ is based on a subset $C$ of $D$
  - The set $C$ is the critical region or rejection region of the past.
  - This corresponding decision rule or test is:
    - reject $H_{0} ⟹$ we accept $H_{a}$ if $(Y_{1}, \dots, Y_{n}) \in C$
    - retain $H_{0} ⟹$ reject $H_{a}$ if $(Y_{1}, \dots, Y_{n}) \notin C$ or $(Y_{1}, \dots, Y_{n}) \in C^{C}$
  - We can't accept the null hypothesis, we can only reject it or fail to reject it (we have insufficient evidence to reject it).
  - This is assuming we have a random sample.
  - The true state of nature is the 'oracle'

&\text{True State of Nature} \
\text{Decision} & H_{0}\text{ is True} & H_{a}\text{ is True} \
\text{Reject }H_{0} & \underbrace{\text{Type I Error}} _{\alpha} & \underbrace{\text{Correct Decision}} _{1-\beta} \
\text{Accept } H*{0} & \underbrace{\text{Correct Decision}} _{1-\alpha} & \underbrace{\text{Type II Error}} _{\beta}
\end

- S i g n i f i c a n c e o f t h e s e e r r o r s : - Y o u h a v e a m a c h i n e l e a r n i n g a l g o t o d e t e c t i f a t u m o r i s m a l i g n a n t o r b e n i g n . - T y p e I e r r o r : y o u s a y i t^{'} s m a l i g n a n t w h e n i t^{'} s a c t u a l l y b e n i g n . T h i s i s b a d b e c a u s e t h e p a t i e n t m i g h t g e t u n n e c e s s a r y t r e a t m e n t . - T y p e I I e r r o r : y o u s a y i t^{'} s b e n i g n w h e n i t^{'} s a c t u a l l y m a l i g n a n t . T h i s i s b a d b e c a u s e t h e p a t i e n t m i g h t n o t g e t n e c e s s a r y t r e a t m e n t . - T h e r e^{'} s a s e e s a w e f f e c t, i f y o u d e c r e a s e t h e p r o b a b i l i t y o f o n e t y p e o f e r r o r, y o u i n c r e a s e t h e p r o b a b i l i t y o f t h e o t h e r t y p e o f e r r o r . - E x a m p l e : - S u p p o s e w e h a v e $ X $ w h i c h i s t h e n u m b e r o f h e a d s . - $ X \sim Bin (n = 3, p) $ - $ H_{0} = p = \frac{1}{2} $ - C o m m o n b e l i e f . - $ H_{a} = p = \frac{2}{3} $ - W e w a n t t o t e s t i f t h e c o i n i s b i a s e d t o w a r d s h e a d s . - W e i g h t e d c o i n . - W h a t v a l u e s o f $ X $ w o u l d y o u a s s i g n t o t h e r e j e c t i o n r e g i o n i f y o u w i s h t o h a v e t h e p r o b a b i l i t y o f a t y p e I e r r o r o f $ α \leq \frac{1}{8} $ a n d t o m i n i m i z e $ β $ c o r r e s p o n d i n g t o t h e v a l u e o f $ α $ s e l e c t e d . - $ P (X = 3) = {(\frac{1}{2})}^{3} = \frac{1}{8} $ - S o w e c a n a s s i g n $ X = 3 $ t o t h e r e j e c t i o n r e g i o n . - S u p p o s e w e g e t t h i s : - $ \begin{matrix} T & T & H \\ H & T & H \end{matrix} $ - W e d o n^{'} t h a v e m u c h e v i d e n c e t o s a y t h e c o i n i s b i a s e d t o w a r d s h e a d s, b e c a u s e w e o n l y g o t 1 a n d 2 h e a d s o u t o f 6 f l i p s . - S u p p o s e w e g e t e x t r e m e c a s e s : - $ \begin{matrix} T & T & T \\ H & H & H \end{matrix} $ - H e r e w e g e t $ 0 $ a n d t h e n $ 3 $ . S o w e h a v e m o r e e v i d e n c e t o s a y t h e c o i n i s b i a s e d t o w a r d s h e a d s, b e c a u s e w e g o t 3 h e a d s o u t o f 6 f l i p s . - O u r c r i t i c a l r e g i o n i s $ X = {0, 3} $ - R e j e c t i o n R e g i o n . - H o w d o w e s o l v e i t m a t h e m a t i c a l l y ? - $ P (X = x) = (\binom{3}{x}) p^{x} (1 - p)^{3 - x} x = 0, 1, 2, 3 $ - C o m p u t e p r o b a b i l i t i e s : - $ \begin{matrix} x & 0 & 1 & 2 & 3 \\ P (X = x | p = \frac{1}{2}) & \frac{1}{8} & \frac{3}{8} & \frac{3}{8} & \frac{1}{8} \\ P (X = x | p = \frac{2}{3}) & \frac{1}{27} & \frac{6}{27} & \frac{12}{27} & \frac{8}{27} \end{matrix} $ - W h a t v a l u e s d o y o u a s s i g n t o r e j e c t i o n i f $ α \leq \frac{1}{8} $ - L o o k a t t h e $ H_{0} $ r o w . - W e s e e t w o v a l u e s $ \leq \frac{1}{8} $ - S o w e s e e $ x = 0, 3 $ - $ x = 0 $ - $ β = P (accept H_{0} | H_{0} is wrong) = P (accept H_{0} | H_{a} is true) $ - $ = P (X \notin RR | p = \frac{2}{3}) $ - $ = P (X \neq 0 | p = \frac{2}{3}) $ - $ = P (X = 1 | p = \frac{2}{3}) + P (X = 2 | p = \frac{2}{3}) + P (X = 3 | p = \frac{2}{3}) $ - $ = \frac{6}{27} + \frac{12}{27} + \frac{8}{27} = \frac{26}{27} $ - $ x = 3 $ - $ β = P (accept H_{0} | H_{a} is true) $ - $ = P (X \notin RR | p = \frac{2}{3}) $ - $ = P (X \neq 3 | p = \frac{2}{3}) $ - $ = P (X = 0 | p = \frac{2}{3}) + P (X = 1 | p = \frac{2}{3}) + P (X = 2 | p = \frac{2}{3}) $ - $ = \frac{1}{27} + \frac{6}{27} + \frac{12}{27} = \frac{19}{27} $ - S u m m a r y - $ $ \begin{matrix} X = 0 & α = \frac{1}{8} & β = \frac{26}{27} & 1 - β = \frac{1}{27} \\ X = 3 & α = \frac{1}{8} & β = \frac{19}{27} & 1 - β = \frac{8}{27} \end{matrix}

- This shows the second rejection region is stronger.
- Example:
- $Y \sim Bin (n = 36, p)$
- $H_{0} : p = 0.5$
- $H_{a} : p = 0.7$
- If it's more than $\frac{1}{2}$ , it's biased.
- Let's say $\frac{22}{36} = \frac{11}{18} = 0.611111111111111$ is the threshold for the rejection region.
- If we choose the critical region to be $| Y - 18 | \geq 4$ .
- What is $α$ and $β$ if $p = 0.7$ ?
- $α = P (reject H_{0} | H_{a} is true)$
- $= P (| Y - 18 | \geq 4 | p = 0.5)$
- $= P (Y \geq 22 | p = 0.5) + P (Y \leq 14 | p = 0.5)$
- Use R
- 1-pbinom(21,36,0.5)+pbinom(14,36,0.5)
- $0.242985$
- Answer from prof $\approx 0.2428$
- $β = P (retain H_{0} | H_{a} is true)$
- $= P (X \notin RR | p = 0.7)$
- $= P (| Y - 18 | < 4 | p = 0.7)$
- $= P (- 3 \leq Y - 18 \leq 3 | p = 0.7)$
- $= P (15 \leq y \leq 21 | p = 0.7)$
- Use R
- pbinom(21,36,0.7)-pbinom(14,36,0.7)
- $0.09155433$
- Answer from prof $\approx 0.0915$
- $β < α$
- Usually we prioritize minimizing $α$ over $β$ , because we want to minimize the probability of making a type I error, which is usually more severe than making a type II error.
- $α < 0.1$ and $β < 0.2$ are normally good.
- Example:
- Large sample test for mean
- $H_{0} : μ = μ_{0}$ where $μ_{0}$ is given
- $H_{a} : μ > μ_{0}$
- Reject $H_{0}$ in favour of $H_{a}$ if $\bar{Y} ≫ μ_{0}$
- Decision rule: reject $H_{0}$ if $\bar{Y} \geq μ_{0} + Z_{α} \frac{σ}{\sqrt{n}}$
- Back to the previous example we can get that $72 + 1.96 \frac{σ}{\sqrt{n}}$ is our point at which we reject.
- Under normality:
- $Y_{1}, \dots, Y_{n} \overset{i i d}{\sim} N (μ, σ^{2})$
- $\frac{\bar{Y} - μ_{0}}{\frac{s}{\sqrt{n}}} \sim t_{(n - 1)}$
- Decision rule: reject $H_{0}$ if $\bar{Y} \geq μ_{0} + t_{(α; n - 1)} \frac{s}{\sqrt{n}}$
- $α = P (T > t_{(α; n - 1)})$
- What if we have two sided alternative hypothesis?
- $H_{0} : μ = μ_{0}$
- $H_{a} : μ \neq μ_{0}$
- Decision rule: reject $H_{0}$ if $| \bar{Y} - μ_{0} | \geq Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}}$
- $Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}} - μ_{0} \leq \bar{Y} \leq Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}} - μ_{0}$
- $α = P (\bar{Y} \leq h \cup \bar{Y} \geq k | H_{0} is true)$
- $= P (\bar{Y} \leq h | H_{0} is true) + P (\bar{Y} \geq k | H_{0} is true)$
- Let the both terms be $\frac{α}{2}$
- Decision rule: reject $H_{0}$ if $| \bar{Y} - μ_{0} | \geq Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}}$
- In case of acceptance:
- Accept $H_{0}$ if $| \bar{Y} - μ_{0} | < Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}}$
- This is the same as the confidence interval for the mean.
- Accept $H_{0}$ if $μ_{0} \in (\bar{Y} - Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}}, \bar{Y} + Z_{\frac{α}{2}} \frac{σ}{\sqrt{n}})$

Next lecture we will learn about P-values.
- Is $\bar{Y}$ sufficiently large to reject $H_{0}$ ?
- Compute the p value, which is the probability of $\bar{Y} \geq \bar{y}$
  - Sample mean is greater than or equal to the observed sample mean, given that $H_{0}$ is true.
  - $p = P (\bar{Y} \geq \bar{y} | H_{0} is true)$ .
  - We expect a small p-value if $H_{a}$ is true, and a large p-value if $H_{0}$ is true.
  - A small p-value is how strongly we reject $H_{0}$ .
  - If $p \leq α$ , we reject $H_{0}$ , otherwise we fail to reject $H_{0}$ .
  - If $α = 0.05$ in finance, then we have $p = 0.048$ we reject $H_{0}$ , but if we have $p = 0.051$ we fail to reject $H_{0}$ , even though the difference is very small.