STA302 Lecture 01

sta302-classical-SLR-assumptions.pdf

• What we finna do?:
  • Datasets like: $(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n)$
  • $X=$ explanatory variable, or a predictor.
  • $Y=$ the response variable.
  • Like input is related to output how?
  • $X,Y$ are numerical variables.
  • Example:
    • Suppose we have some data points.
  • Goals:
    • Summarize the relationship between the explanatory and response.
      • Start with a simple linear model.
      • We want to find the line of best fit.
        • Keep in mind overfitting.
      • Criteria:
        • Least squares estimator
        • Maximum Likelihood Estimator
    • Hypothesis Testing
      • Model:
        • Suppose we have a model like so
          • $y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$
          • Price of unknown car $x_i$ is predicted by $y_i$ some unknown mileage.
        • Estimates
          • ${\hat{y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$
        • Hypo test
          • $H_0:{\beta }_1=0$ vs $H_a:{\beta }_1\ne 0$
          • Are these things even related?
            • If not related, then just use $\overline{y}$, the avg.
    • If $H_0$ is rejected (useful model), then we try to make a CI to estimate $\mathrm{E}[Y]$
      • Lots of data to predict the average
    • If $H_0$ is rejected, then provide a prediction for $Y_{\ast }$
      • One piece of data to predict another single piece of data
      • More variability for this case.
      • Simple linear regression model.
        • SLR
    • Generalize to model with several predictors
      • Multiple linear regression
        • MLR
      • Like adding '# of previous owners' and 'colour' and such to the previous example model
      • Use simpler models if we get the same result
• Regression Model with Conditional Distributions
  • seq is a sequence of elements.
  • Types of regression:
    • If $P(Y|X)$
      • normal = classical
      • poisson = poisson
      • … = …
    • MLE and Least Squares type estimators have nice formulas.
    • But the other types don't exactly have those.
  • We don't say the regression model came from the data. We say that the regression model created the data, we're just trying to find the things which produce that data.
• Classical SLR model
  • $Y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$
  • Assuming ${\epsilon }_i\stackrel{\text{iid}}{\sim }N(0,{\sigma }^2)$
  • $\mathrm{E}[Y_i]=\mathrm{E}[{\beta }_0]+\mathrm{E}[{\beta }_1{x}_i]+\mathrm{E}\left[{\epsilon }_i\right]$
  • $\mathrm{E}[Y_i]={\beta }_0+{\beta }_1{x}_i$
  • $\mathrm{Var}(Y_i)={\sigma }^2$ #tk prac deriv
  • We can't use a linear model in this case
  • Before fitting, make scatter plot and see if it's reasonable to use one.
  • This can work:
  • Here you can see that the variance increases as we go, non-constant variance.
  • LINE assumptions:
    • L
      • Linear
      • Expected value is a linear function
    • I
      • Errors are independent
    • N
      • Errors are normal
    • E
      • Errors have equal variance
      • Also referred to as "homoscedasticity"?
  • In the previous model we have these assumptions.
• Understand regression models with simulation
  • Produce widgets
  • $Y=$ prod cost
  • $X=$ widgets
  • 
  • Then you can guess ${\beta }_0,{\beta }_1$, then simulate and see if the scatter plot looks similar.
  • We can use least square estimation to find these.
• Least squares estimation
  • For ${\beta }_0$ and ${\beta }_1$
  • Minimize with respect to ${\beta }_0,{\beta }_1$
  • We have our observations: $[y_1-({\beta }_0+{\beta }_1{x}_1){]}^2+\cdots +[y_n-({\beta }_0+{\beta }_1{x}_n){]}^2$
  • We minimize that.
  • $\stackrel{min}{{\beta }_0,{\beta }_1}\sum _{i=1}^n[y_i-({\beta }_0+{\beta }_1{x}_i){]}^2$
  • Simplify further
  • $\stackrel{min}{{\beta }_0,{\beta }_1}Q$
  • $Q$ is a quadratic function, which is our function we wanted to minimize.
  • Diff and set to 0?? Crit vals?
    • K
    • Part diffs 🙏
    • $Q(y,x,{\beta }_0,{\beta }_1)$
    • $\frac{\partial Q}{\partial {\beta }_0}\frac{\text{differential}{\beta }_0}{\text{differential}x}+\frac{\partial Q}{\partial {\beta }_1}\frac{\text{differential}{\beta }_1}{\text{differential}x}$
    • ?
    • $\frac{\partial Q}{\partial {\beta }_0}=\sum _{i=1}^n-2[y_i-{\beta }_0-{\beta }_1{x}_i]$
    • $\frac{\partial Q}{\partial {\beta }_1}=\sum _{i=1}^n-2[y_i-{\beta }_0-{\beta }_1{x}_i]x_i$
  • Teach steppin in
    • Find $\frac{\partial Q}{\partial {\beta }_0}$ and $\frac{\partial Q}{\partial {\beta }_1}$
    • I think I did
    • Then solve the system of equations.
    • 1: $\frac{\partial Q}{\partial {\beta }_0}=0$
    • 2: $\frac{\partial Q}{\partial {\beta }_1}=0$
    • $\frac{\partial Q}{\partial {\beta }_0}=\sum _{i=1}^n-2[y_i-{\beta }_0-{\beta }_1{x}_i]$
      • $\frac{\partial Q}{\partial {\beta }_0}=-2\sum _{i=1}^n[y_i-{\beta }_0-{\beta }_1{x}_i]$
      • $\frac{\partial Q}{\partial {\beta }_0}=-2[\sum _{i=1}^n{y}_i-n{\beta }_0-{\beta }_1\sum _{i=1}^n{x}_i]$
      • $\frac{\partial Q}{\partial {\beta }_0}=-2[n\overline{y}-n{\beta }_0-{\beta }_{1}n\overline{x}]$
      • $0=-2[n\overline{y}-n{\beta }_0-{\beta }_{1}n\overline{x}]$
      • $0=n\overline{y}-n{\beta }_0-{\beta }_{1}n\overline{x}$
      • $0=\overline{y}-{\beta }_0-{\beta }_1\overline{x}$
      • $\overline{y}={\beta }_0+{\beta }_1\overline{x}$
    • $\frac{\partial Q}{\partial {\beta }_1}=\sum _{i=1}^n-2[y_i-{\beta }_0-{\beta }_1{x}_i]x_i$
      • $\frac{\partial Q}{\partial {\beta }_1}=-2\sum _{i=1}^n[y_i-{\beta }_0-{\beta }_1{x}_i]x_i$
      • $\frac{\partial Q}{\partial {\beta }_1}=-2\sum _{i=1}^n[y_i{x}_i-{\beta }_0{x}_i-{\beta }_1{x}_i^2]$
      • $\frac{\partial Q}{\partial {\beta }_1}=-2[\sum _{i=1}^n{y}_i{x}_i-{\beta }_0\sum _{i=1}^n{x}_i-{\beta }_1\sum _{i=1}^n{x}_i^2]$
      • $0=-2[\sum _{i=1}^n{y}_i{x}_i-{\beta }_0\sum _{i=1}^n{x}_i-{\beta }_1\sum _{i=1}^n{x}_i^2]$
      • $0=\sum _{i=1}^n{y}_i{x}_i-{\beta }_0\sum _{i=1}^n{x}_i-{\beta }_1\sum _{i=1}^n{x}_i^2$
      • $\sum _{i=1}^n{y}_i{x}_i={\beta }_0\sum _{i=1}^n{x}_i+{\beta }_1\sum _{i=1}^n{x}_i^2$
      • $\sum _{i=1}^n{y}_i{x}_i={\beta }_{0}n\overline{x}+{\beta }_1\sum _{i=1}^n{x}_i^2$
  • Normal equations
    • 1: $n{\beta }_0+{\beta }_1\sum _{i=1}^n{x}_i=\sum _{i=1}^n{y}_i$
    • 2: ${\beta }_0\sum _{i=1}^n{x}_i+{\beta }_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{y}_i{x}_i$
    • Find ${\beta }_1$ using Cramer's rule and ${\beta }_0$ with substitution.
      • $\mathrm{\Delta }=$ deteminant
      • ${\beta }_1=\frac{\mathrm{\Delta }{\beta }_1}{\mathrm{\Delta }}={\displaystyle \frac{\left|\begin{array}{cc}n& \sum _{i=1}^n{y}_i\\ \sum _{i=1}^n{x}_i& \sum _{i=1}^n{x}_i{y}_i\end{array}\right|}{\left|\begin{array}{cc}n& \sum _{i=1}^n{x}_i\\ \sum _{i=1}^n{x}_i& \sum _{i=1}^n{x}_i^2\end{array}\right|}}$
        • How do we get this?
        • You can multiply by inverse and other methods too.
        • Otherwise, we want ${\hat{\beta }}_1$.
        • Denominator is $\mathrm{\Delta }$ of the system. The coeffs of the system from eq 1.
        • $\mathrm{\Delta }{\beta }_1$ is the coeffs of the eq 2.
        • #tk
          • Find ${\hat{\beta }}_0$ using Cramer's rule
          • Show it's the same as $\overline{y}-{\hat{\beta }}_1\overline{x}$
          • ${\hat{\beta }}_0=\frac{\left|\begin{array}{cc}\sum _{i=1}^n{y}_i& \sum _{i=1}^n{x}_i\\ \sum _{i=1}^n{x}_i{y}_i& \sum _{i=1}^n{x}_i^2\end{array}\right|}{\mathrm{\Delta }}=\frac{\sum _{i=1}^n{y}_i\sum _{i=1}^n{x}_i^2-\sum _{i=1}^n{x}_i\sum _{i=1}^n{x}_i{y}_i}{n\sum _{i=1}^n{x}_i^2-{\left(\sum _{i=1}^n{x}_i\right)}^2}$
      • Compute the deteminant.
      • ${\hat{\beta }}_1=\frac{\mathrm{\Delta }{\beta }_1}{\mathrm{\Delta }}=\frac{n\sum _{i=1}^n{x}_i{y}_i-\left(\sum _{i=1}^n{x}_i\right)\left(\sum _{i=1}^n{y}_i\right)}{n\sum _{i=1}^n{x}_i^2-{\left(\sum _{i=1}^n{x}_i\right)}^2}$
      • I remember something about ${\hat{\beta }}_1=\frac{S_{XY}}{S_{XX}}$ #tk
    • Find ${\beta }_0$
      • 1: $n{\beta }_0+{\hat{\beta }}_1\sum _{i=1}^n{x}_i=\sum _{i=1}^n{y}_i$
      • $n{\beta }_0=\sum _{i=1}^n{y}_i-{\hat{\beta }}_1\sum _{i=1}^n{x}_i$
      • ${\beta }_0=\frac{1}{n}\sum _{i=1}^n{y}_i-\frac{{\hat{\beta }}_1}{n}\sum _{i=1}^n{x}_i$
      • ${\beta }_0=\overline{y}-{\hat{\beta }}_1\overline{x}$
        • STA258
    • Test problem 1:
      • $n=10$
      • $\sum _{i=1}^n{y}_i=15$
      • $\sum _{i=1}^n{y}_i^2=121$
      • $\sum _{i=1}^n{x}_i=2$
      • $\sum _{i=1}^n{x}_i^2=180$
      • $\sum _{i=1}^n{x}_i{y}_i=98$
      • Find estimates for ${\beta }_0$ and ${\beta }_1$ explicitly
      • ${\beta }_0=\overline{y}-{\beta }_1\overline{x}$
        • #tk also memorize this jawn
      • ${\beta }_1=\frac{n\sum _{i=1}^n{x}_i{y}_i-\left(\sum _{i=1}^n{x}_i\right)\left(\sum _{i=1}^n{y}_i\right)}{n\sum _{i=1}^n{x}_i^2-{\left(\sum _{i=1}^n{x}_i\right)}^2}$
        • #tk memorize this jawn
      • ${\beta }_1=\frac{(10)(98)-(2)(15)}{(10)(180)-(2{)}^2}=\frac{475}{898}=0.528953229398664$
      • ${\beta }_0=\frac{1}{n}\sum _{i=1}^n{y}_i-\frac{475}{898}\frac{1}{n}\sum _{i=1}^n{x}_i$
      • ${\beta }_0=\frac{1}{10}(15)-\frac{475}{898}\frac{1}{10}(2)=\frac{626}{449}=1.39420935412027$
      • So we can have $y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$
      • $y_i=\frac{626}{449}+\frac{475}{898}{x}_i+{\epsilon }_i$
    • Soln:
      • $10{\beta }_0+2{\beta }_1=15$
      • $2{\beta }_0+180{\beta }_1=98$
      • 5x $10{\beta }_0+900{\beta }_1=490$
      • 1-2
      • $(10{\beta }_0)-(10{\beta }_0)+(2{\beta }_1)-(900{\beta }_1)=(15)-(490)$
      • $-898{\beta }_1=-475$
      • $898{\beta }_1=475$
      • ${\beta }_1=\frac{475}{898}=0.528953229398664$
      • Same answer.
      • Or cramers
      • ${\hat{\beta }}_0=\frac{\mathrm{\Delta }{\beta }_0}{\mathrm{\Delta }}=\frac{\left|\begin{array}{cc}10& 2\\ 98& 180\end{array}\right|}{\left|\begin{array}{cc}10& 2\\ 2& 180\end{array}\right|}=\frac{(15)(180)-(2)(98)}{(10)(180)-(2)(2)}=\frac{2504}{1796}=1.3942093541$
      • same answer.
      • ${\hat{\beta }}_1=\frac{950}{1796}=\frac{475}{898}=0.528953229398664$
      • We can also do:
        • $\left|\begin{array}{cc}10& 2\\ 2& 180\end{array}\right|\left|\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right|=\left|\begin{array}{c}15\\ 98\end{array}\right|$
    • #tk homework:
      • $A=\left|\begin{array}{cc}a& b\\ c& d\end{array}\right|$ what's the formula for $A^{-1}$?
    • Proof: ${\hat{\beta }}_1=\frac{S_{XY}}{S_{XX}}=\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}$ is the same as the previous jawn with cramers.
      • $\frac{\sum _{i=1}^n(x_i{y}_i-x_i\overline{y}-\overline{x}{y}_i+\overline{x}\overline{y})}{\sum _{i=1}^n(x_i^2-2x_i\overline{x}+{\overline{x}}^2)}$
      • $\frac{\sum _{i=1}^n{x}_i{y}_i-\overline{y}\sum _{i=1}^n{x}_i-\overline{x}\sum _{i=1}^n{y}_i+n\overline{x}\overline{y}}{\sum _{i=1}^n{x}_i^2-2\overline{x}\sum _{i=1}^n{x}_i+n{\overline{x}}^2}$
      • $\frac{\sum _{i=1}^n{x}_i{y}_i-\frac{1}{n}\sum _{i=1}^n{y}_i\sum _{i=1}^n{x}_i-\frac{1}{n}\sum _{i=1}^n{x}_i\sum _{i=1}^n{y}_i+n\overline{x}\overline{y}}{\sum _{i=1}^n{x}_i^2-2\overline{x}\sum _{i=1}^n{x}_i+n{\overline{x}}^2}$
      • $S_{XY}=\sum _{i=1}^n{x}_i{y}_i-\frac{n\overline{y}}{n}\sum _{i=1}^n{x}_i-\frac{n\overline{x}\sum _{i=1}^n{y}_i}{n}+n\overline{x}\overline{y}$
      • $S_{XY}=\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}$
      • $n{S}_{XY}=n\sum _{i=1}^n{x}_i{y}_i-n^2\overline{x}\overline{y}$
      • $n{S}_{XY}=n\sum _{i=1}^n{x}_i{y}_i-n^2\frac{1}{n}\sum _{i=1}^n{x}_i\frac{1}{n}\sum _{i=1}^n{y}_i$
      • $n{S}_{XY}=n\sum _{i=1}^n{x}_i{y}_i-n^2\frac{1}{n^2}\sum _{i=1}^n{x}_i\sum _{i=1}^n{y}_i$
      • $n{S}_{XY}=n\sum _{i=1}^n{x}_i{y}_i-\sum _{i=1}^n{x}_i\sum _{i=1}^n{y}_i$
      • Note that for $S_{XX}$ we have would have $\sum _{i=1}^n{x}_i{y}_i$ as $\sum _{i=1}^n{x}_i^2$.
      • If we extrapolate that we see that we basically have the same proof for the bottom.
      • So $\frac{n{S}_{XY}}{n{S}_{XX}}=\frac{S_{XX}}{S_{XY}}={\hat{\beta }}_1$
  • Example:
    • Cool shit: s.xy=(x-x.bar)%*%(y-y.bar);
    • That will do the matrix multiplication in R.
  • Maximum Likelihood Estimator for ${\beta }_0,{\beta }_1$
    • $f(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$ for $-\mathrm{\infty }<x<\mathrm{\infty }$, $\mu \in \mathbb{R}$, ${\sigma }^2>0$
    • SLR
    • $y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i$
      • ${\epsilon }_i\stackrel{iid}{\sim }N(0,{\sigma }^2)$
    • $\mathrm{E}[y_i]={\beta }_0+{\beta }_1{x}_i$
    • $\mathrm{Var}(y_i)={\sigma }^2$
    • Likelihood for one observation of $y_i$
      • The PDF of that observation
    • $L(y_i|{\beta }_0,{\beta }_1,{\sigma }^2)=f(y_i)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{[y_i-{\beta }_0-{\beta }_1{x}_i{]}^2}{2{\sigma }^2}}$
      • You can show that constant plus normal is normal with mgf method, simple.
    • Likelihood for random samples.
      • $y_1,\dots ,y_n$
      • Take products.
    • $L(y_1,\dots ,y_n|{\beta }_0,{\beta }_1,{\sigma }^2)=\prod _{i=1}^{n}f(y_i)$