STA258 Lecture 20

• Regression Analysis.
  • Example:
    • Suppose you have $x$ which is a predictor.
      • Weight of human
    • $y$ is your response type.
      • Are they male or female
    • Suppose we have at $50\mathrm{kg}$ we have a response at $0.2$, etc.
  • Regression wants us to find a line of best fit.
    • Lets us assume a set of points of observations $\{(x_i,y_i){\}}_{i=1}^n$ is given.
    • Assumes the predictors and response variables are real-valued.
  • What's the line of fit?
    • $y_i=\alpha +\beta x_i+{ϵ}_{i}i=1,\dots ,n$
    • If we have $n$ observations, what are the optimal values of $\alpha$ and $\beta$?
    • Estimates are: $\hat{\alpha }$ and $\hat{\beta }$
  • Ordinary least squares (OLS) is the most common method to find the line of best fit.
  • Minimize the sum of squared residuals:
    • $\sum _{i=1}^n(y_i-\hat{\alpha }-\hat{\beta }{x}_i{)}^2$
    • $E=E(\alpha ,\beta )=\sum _{i=1}^n(y_i-\alpha -\beta x_i{)}^2$
    • We need to differentiate $E$
    • $\alpha$
      • $\frac{\partial E}{\partial \alpha }=\frac{\partial }{\partial \alpha }\sum _{i=1}^n[y_i-\alpha -\beta x_i{]}^2$
      • $\frac{\partial E}{\partial \alpha }=\sum _{i=1}^n\frac{\partial }{\partial \alpha }[y_i-\alpha -\beta x_i{]}^2$
      • $\frac{\partial E}{\partial \alpha }=\sum _{i=1}^n\frac{\partial }{\partial \alpha }2[y_i-\alpha -\beta x_i](-1)$
      • $\frac{\partial E}{\partial \alpha }=-2\sum _{i=1}^n(y_i-\alpha -\beta x_i)$
      • $\frac{\partial E}{\partial \alpha }=0$
      • $0=-2\sum _{i=1}^n(y_i-\alpha -\beta x_i)$
      • $0=\sum _{i=1}^n{y}_i-n\alpha -\beta \sum _{i=1}^n{x}_i$
      • $0=\overline{y}-\alpha -\beta \overline{x}$
      • $\hat{\alpha }=\overline{y}-\beta \overline{x}$
      • So first we need to find $\beta$. Then we can estimate $\alpha$.
    • $\beta$
      • $\frac{\partial E}{\partial \beta }=\frac{\partial }{\partial \beta }\sum _{i=1}^n[y_i-\alpha -\beta x_i{]}^2$
      • $\frac{\partial E}{\partial \beta }=\sum _{i=1}^n\frac{\partial }{\partial \beta }[y_i-\alpha -\beta x_i{]}^2$
      • $\frac{\partial E}{\partial \beta }=\sum _{i=1}^n\frac{\partial }{\partial \beta }2[y_i-\alpha -\beta x_i](-x_i)$
      • $\frac{\partial E}{\partial \beta }=-2\sum _{i=1}^n(y_i-\alpha -\beta x_i)x_i$
      • $\frac{\partial E}{\partial \beta }=0$
      • $0=-2\sum _{i=1}^n(y_i-\alpha -\beta x_i)x_i$
      • $0=\sum _{i=1}^n{y}_i{x}_i-\alpha \sum _{i=1}^n{x}_i-\beta \sum _{i=1}^n{x}_i^2$
      • $\beta n{\overline{x}}^2-\beta \sum _{i=1}^n{x}_i^2=-\sum _{i=1}^n{x}_i{y}_i+n\overline{x}\overline{y}$
      • $\beta (\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2)=\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}$
      • $\hat{\beta }=\frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}$
      • $\hat{\alpha }=\overline{y}-\hat{\beta }\overline{x}$
      • We know that $\sum _{i=1}^n(x_i-\overline{x}{)}^2=\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2$
      • We can get that $\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})=\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}$
        • $\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})=\sum _{i=1}^n(x_i{y}_i-\overline{x}{y}_i-\overline{y}{x}_i+\overline{x}\overline{y})$
        • $\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})=\sum _{i=1}^n{x}_i{y}_i-\overline{x}\sum _{i=1}^n{y}_i-\overline{y}\sum _{i=1}^n{x}_i+\sum _{i=1}^n\overline{x}\overline{y}$
        • $\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})=\sum _{i=1}^n{x}_i{y}_i-\overline{x}(n\overline{y})-\overline{y}(n\overline{x})+n\overline{x}\overline{y}$
        • $\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})=\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}$
      • So $\hat{\beta }=\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}$
  • We can also do this with matrices. Hessian Matrix and such.
    • Use some elementary results from MAT223 and matrix calculus.
• Example:
  • We have $y_i={\beta }_1{x}_{i_1}+{\beta }_2{x}_{i2}+\dots {\beta }_p{x}_{ip}+{ϵ}_i$ for $i=1,\dots ,n$
  • We have $p$ predictors.
  • $\overrightarrow{y}={\left[\begin{array}{c}{y}_1\\ y_2\\ \dots \\ y_n\end{array}\right]}_{n\ge 1}$
  • $X$ is a design matrix. This has all observations.
    • $X=\left[\begin{array}{cccc}{X}_{11}& X_{12}& \dots & X_{1p}\\ \dots & \dots & \dots & \dots \\ X_{i1}& X_{i2}& \dots & X_{ip}\end{array}\right]$
  • $\overrightarrow{\beta }=\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\\ \dots \\ {\beta }_n\end{array}\right]$
  • $\overrightarrow{ϵ}=\left[\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ \dots \\ {ϵ}_n\end{array}\right]$
  • $\overrightarrow{Y}=X\overrightarrow{\beta }+\overrightarrow{ϵ}$
  • This means $\overrightarrow{ϵ}=\overrightarrow{Y}-X\overrightarrow{\beta }$
  • ${\left|\overrightarrow{v}\right|}_2^1=\sqrt{v_1^2+v_2^2+\cdots +v_n^2}$
  • ${\left|\overrightarrow{v}\right|}_2^2=V^{T}V$
    • $=\left[\begin{array}{cccc}{V}_1& V_2& \dots & V_n\end{array}\right]\left[\begin{array}{c}{V}_1\\ V_2\\ \dots \\ V_n\end{array}\right]$
    • $=V_1^2+V_2^2+\cdots +V_n^2$
  • $min{|\overrightarrow{Y}-X\overrightarrow{\beta }|}_2^2$
  • We want to minimize with respect to $\overrightarrow{\beta }$.
  • $\mathrm{E}[\overrightarrow{\beta }]={|\overrightarrow{Y}-X\overrightarrow{\beta }|}_2^2=(\overrightarrow{Y}-X\overrightarrow{\beta }{)}^T(\overrightarrow{Y}-X\overrightarrow{\beta })$
    • $=(Y^T-{\beta }^T{X}^T)(\overrightarrow{Y}-X\overrightarrow{\beta })$
    • $=Y^{T}Y-Y^{T}X\beta -{\beta }^T{X}^{T}Y+{\beta }^T{X}^{T}X\beta$
    • $Y^{T}X\beta$
      • $Y^{T}X\beta$ is a scalar. So it's equal to its transpose.
      • $={\beta }^T{X}^{T}Y$
    • $=Y^{T}Y-2{\beta }^T{X}^{T}Y+{\beta }^T{X}^{T}X\beta$
    • Now we need to differentiate between the scalars and vectors.
    • We'll use two identities from matrix calculus.
      • $A\in {\mathbb{R}}^{P\times P}$
      • ${\mathrm{\nabla }}_{\beta }({\beta }^{T}A\beta )=(A\times A^T)\beta$
      • ${\mathrm{\nabla }}_{\beta }({\beta }^T\overrightarrow{A})=\overrightarrow{A}$
      • If $A$ is symmetric then $A=A^T$ and ${\mathrm{\nabla }}_{\beta }({\beta }^{T}A\beta )=2A\beta$
    • So ${\mathrm{\nabla }}_{\beta }E(\beta )=2X^{T}X\beta -2X^{T}Y=0$
      • $X^{T}X\beta =X^{T}Y$
      • To isolate $\beta$
      • $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$
      • Same estimates as before, but now we can do this with more predictors.