🛠 toolkit for comparing models

🛠 toolkit for comparing models

👉 F-test

🛠 F-test for Multiple Linear Regression

• Comparing the full model to the intercept only model

🛠 F-test for Multiple Linear Regression

• Comparing the full model to the intercept only model
• $H_0:{\beta }_1={\beta }_2=\cdots ={\beta }_k=0$

🛠 F-test for Multiple Linear Regression

• $F=\frac{MSModel}{MSE}$

🛠 F-test for Multiple Linear Regression

• $F=\frac{MSModel}{MSE}$
• df for the Model?

🛠 F-test for Multiple Linear Regression

• $F=\frac{MSModel}{MSE}$
• df for the Model?
  • k

🛠 F-test for Multiple Linear Regression

• $F=\frac{MSModel}{MSE}$
• df for the Model?
  • k
• df for the errors?

🛠 Nested F-test for Multiple Linear Regression

• What does "nested" mean?
  • You have a "small" model and a "large" model where the "small" model is completely contained in the "large" model

🛠 Nested F-test for Multiple Linear Regression

• What does "nested" mean?
  • You have a "small" model and a "large" model where the "small" model is completely contained in the "large" model
• The F-test we have learned so far is one example of this, comparing:
  • $y={\beta }_0+ϵ$ (small)
  • $y={\beta }_0+{\beta }_1+{\beta }_2+\cdots +{\beta }_k+ϵ$ (large)

🛠 Nested F-test for Multiple Linear Regression

• The full (large) model has $k$ predictors, the reduced (small) model has $k-p$ predictors

🛠 Nested F-test for Multiple Linear Regression

• The full (large) model has $k$ predictors, the reduced (small) model has $k-p$ predictors
• What is $H_0$?

🛠 Nested F-test for Multiple Linear Regression

• The full (large) model has $k$ predictors, the reduced (small) model has $k-p$ predictors
• What is $H_0$?
  • $H_0:$ ${\beta }_i=0$ for all $p$ predictors being dropped from the full model

🛠 Nested F-test for Multiple Linear Regression

• The full (large) model has $k$ predictors, the reduced (small) model has $k-p$ predictors
• What is $H_0$?
  • $H_0:$ ${\beta }_i=0$ for all $p$ predictors being dropped from the full model
• What is $H_A$?

🛠 Nested F-test for Multiple Linear Regression

• The full (large) model has $k$ predictors, the reduced (small) model has $k-p$ predictors
• What is $H_0$?
  • $H_0:$ ${\beta }_i=0$ for all $p$ predictors being dropped from the full model
• What is $H_A$?
  • $H_A:$ ${\beta }_i\ne 0$ for at least one of the $p$ predictors dropped from the full model

🛠 Nested F-test for Multiple Linear Regression

• Which of these are nested models?

(1) $y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$
(2) $y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3{x}_1\ast x_2+ϵ$
(3) $y={\beta }_0+{\beta }_1{x}_3+ϵ$
(4) $y={\beta }_0+{\beta }_2{x}_2+ϵ$
(5) $y={\beta }_0+{\beta }_1{x}_4+ϵ$

🛠 Nested F-test for Multiple Linear Regression

• Comparing these two models, what is $p$?

(1) $y={\beta }_0+{\beta }_2{x}_2+ϵ$
(2) $y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3{x}_1\ast x_2+ϵ$

🛠 Nested F-test for Multiple Linear Regression

• Comparing these two models, what is $p$?

(1) $y={\beta }_0+{\beta }_2{x}_2+ϵ$
(2) $y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3{x}_1\ast x_2+ϵ$

• $p=2$

🛠 Nested F-test for Multiple Linear Regression

• Comparing these two models, what is $p$?

(1) $y={\beta }_0+{\beta }_2{x}_2+ϵ$
(2) $y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_3{x}_1\ast x_2+ϵ$

• $p=2$
• What is $k$?

🛠 Nested F-test for Multiple Linear Regression

• Goal: Trying to predict the weight of fish based on their length and width

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

🛠 Nested F-test for Multiple Linear Regression

• Goal: Trying to predict the weight of fish based on their length and width

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

• What is the equation for model1?

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

• If we want to do a nested F-test, what is $H_0$?

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

• If we want to do a nested F-test, what is $H_0$?
  • $H_0:{\beta }_3={\beta }_4=0$

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

• If we want to do a nested F-test, what is $H_0$?
  • $H_0:{\beta }_3={\beta }_4=0$
• What is $H_A$?

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

• If we want to do a nested F-test, what is $H_0$?
  • $H_0:{\beta }_3={\beta }_4=0$
• What is $H_A$?
  • $H_A:{\beta }_3\ne 0$ or ${\beta }_4\ne 0$

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + I(Length ^ 2) + I(Width ^ 2) + Length * Width,
  data = Perch
  )

• If we want to do a nested F-test, what is $H_0$?
  • $H_0:{\beta }_3={\beta }_4=0$
• What is $H_A$?
  • $H_A:{\beta }_3\ne 0$ or ${\beta }_4\ne 0$
• What are the degrees of freedom of this test? (n = 56)

🛠 Nested F-test for Multiple Linear Regression

• $F=\frac{SSMODE{L}_{Full}-SSMODE{L}_{Reduced}/p}{SS{E}_{Full}/n-k-1}$

🛠 Nested F-test for Multiple Linear Regression

• $F=\frac{SSMODE{L}_{Full}-SSMODE{L}_{Reduced}/p}{SS{E}_{Full}/n-k-1}$
• $SSMODE{L}_{Full}-SSMODE{L}_{Reduced}$:

SSModel2 - SSModel1

## [1] 8765

🛠 Nested F-test for Multiple Linear Regression

• $F=\frac{SSMODE{L}_{Full}-SSMODE{L}_{Reduced}/p}{SS{E}_{Full}/n-k-1}$

🛠 Nested F-test for Multiple Linear Regression

• $F=\frac{SSMODE{L}_{Full}-SSMODE{L}_{Reduced}/p}{SS{E}_{Full}/n-k-1}$

((SSModel2 - SSModel1) / 2) /
  1860

## [1] 2.36

🛠 Nested F-test for Multiple Linear Regression

• $F=\frac{SSMODE{L}_{Full}-SSMODE{L}_{Reduced}/p}{SS{E}_{Full}/n-k-1}$

((SSModel2 - SSModel1) / 2) /
  1860

## [1] 2.36

• What are the degrees of freedom for this test?

🛠 Nested F-test for Multiple Linear Regression

• $F=\frac{SSMODE{L}_{Full}-SSMODE{L}_{Reduced}/p}{SS{E}_{Full}/n-k-1}$

((SSModel2 - SSModel1) / 2) /
  1860

## [1] 2.36

• What are the degrees of freedom for this test?
  • 2, 50

🛠 $R^2$ for Multiple Linear Regression

🛠 $R^2$ for Multiple Linear Regression

• $R^2=\frac{SSModel}{SSTotal}$

🛠 $R^2$ for Multiple Linear Regression

• $R^2=\frac{SSModel}{SSTotal}$
• $R^2=1-\frac{SSE}{SSTotal}$

🛠 $R^2$ for Multiple Linear Regression

• $R^2=\frac{SSModel}{SSTotal}$
• $R^2=1-\frac{SSE}{SSTotal}$
• As is, if you add a predictor this will always increase. Therefore, we have $R_{adj}^2$ that has a small "penalty" for adding more predictors

🛠 $R^2$ for Multiple Linear Regression

• $R^2=\frac{SSModel}{SSTotal}$
• $R^2=1-\frac{SSE}{SSTotal}$
• As is, if you add a predictor this will always increase. Therefore, we have $R_{adj}^2$ that has a small "penalty" for adding more predictors
• $R_{adj}^2=1-\frac{SSE/(n-k-1)}{SSTotal/(n-1)}$

🛠 $R^2$ for Multiple Linear Regression

• $R^2=\frac{SSModel}{SSTotal}$
• $R^2=1-\frac{SSE}{SSTotal}$
• As is, if you add a predictor this will always increase. Therefore, we have $R_{adj}^2$ that has a small "penalty" for adding more predictors
• $R_{adj}^2=1-\frac{SSE/(n-k-1)}{SSTotal/(n-1)}$
• $\frac{SSTotal}{n-1}=\frac{\sum (y-\overline{y}{)}^2}{n-1}$ What is this?

🛠 $R^2$ for Multiple Linear Regression

• $R^2=\frac{SSModel}{SSTotal}$
• $R^2=1-\frac{SSE}{SSTotal}$
• As is, if you add a predictor this will always increase. Therefore, we have $R_{adj}^2$ that has a small "penalty" for adding more predictors
• $R_{adj}^2=1-\frac{SSE/(n-k-1)}{SSTotal/(n-1)}$
• $\frac{SSTotal}{n-1}=\frac{\sum (y-\overline{y}{)}^2}{n-1}$ What is this?
  • Sample variance! $S_Y^2$

🛠 $R_{adj}^2$ for Multiple Linear Regression

glance(model1)

## # A tibble: 1 x 11
##   r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
##       <dbl>         <dbl> <dbl>     <dbl>    <dbl> <int>  <dbl> <dbl> <dbl>
## 1     0.985         0.984  44.2     1115. 3.75e-47     4  -290.  589.  599.
## # … with 2 more variables: deviance <dbl>, df.residual <int>

glance(model2)

## # A tibble: 1 x 11
##   r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
##       <dbl>         <dbl> <dbl>     <dbl>    <dbl> <int>  <dbl> <dbl> <dbl>
## 1     0.986         0.985  43.1      705. 4.41e-45     6  -287.  588.  602.
## # … with 2 more variables: deviance <dbl>, df.residual <int>

Model Comparision criteria

• We are looking for reasonable ways to balance "goodness of fit" (how well the model fits the data) with "parsimony"

Model Comparision criteria

• We are looking for reasonable ways to balance "goodness of fit" (how well the model fits the data) with "parsimony"
• $R_{adj}^2$ gets at this by adding a penalty for adding variables

Log Likelihood

• Both AIC and BIC are calculated using the log likelihood

$\log \left(L\right)=-\frac{n}{2}\left[\log \right(2\pi )+\log (SSE/n)+1]$

Log Likelihood

• Both AIC and BIC are calculated using the log likelihood

$\log \left(L\right)=-\frac{n}{2}\left[\log \right(2\pi )+\log (SSE/n)+1]$

• $\log ={\log }_e$, log() in R

Log Likelihood

• Both AIC and BIC are calculated using the log likelihood

$\log \left(L\right)=-\frac{n}{2}\left[\log \right(2\pi )+\log (SSE/n)+1]$

• $\log ={\log }_e$, log() in R

glance(model1)

## # A tibble: 1 x 11
##   r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
##       <dbl>         <dbl> <dbl>     <dbl>    <dbl> <int>  <dbl> <dbl> <dbl>
## 1     0.985         0.984  44.2     1115. 3.75e-47     4  -290.  589.  599.
## # … with 2 more variables: deviance <dbl>, df.residual <int>

-56 / 2 * (log(2 * pi) + log(101765 / 56) + 1)

## [1] -290

AIC

• Akaike's Information Criterion
• $AIC=2(k+1)-2\log \left(L\right)$
• $k$ is the number of predictors in the model
• lower AIC values are better

BIC

• Bayesian Information Criterion
• $BIC=\log \left(n\right)(k+1)-2\log \left(L\right)$
• $k$ is the number of predictors in the model
• lower BIC values are better

AIC and BIC can disagree!

glance(model1)

## # A tibble: 1 x 11
##   r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
##       <dbl>         <dbl> <dbl>     <dbl>    <dbl> <int>  <dbl> <dbl> <dbl>
## 1     0.985         0.984  44.2     1115. 3.75e-47     4  -290.  589.  599.
## # … with 2 more variables: deviance <dbl>, df.residual <int>

glance(model2)

## # A tibble: 1 x 11
##   r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC
##       <dbl>         <dbl> <dbl>     <dbl>    <dbl> <int>  <dbl> <dbl> <dbl>
## 1     0.986         0.985  43.1      705. 4.41e-45     6  -287.  588.  602.
## # … with 2 more variables: deviance <dbl>, df.residual <int>

• the penalty term is larger in BIC than in AIC