Multiple Logistic Regression
1 / 23
by Dr. Lucy D'Agostino McGowan
2 / 23
by Dr. Lucy D'Agostino McGowan
Types of statistical models
| response | predictor(s) | model |
|---|---|---|
| quantitative | one quantitative | simple linear regression |
| quantitative | two or more (of either kind) | multiple linear regression |
| binary | one (of either kind) | simple logistic regression |
| binary | two or more (of either kind) | multiple logistic regression |
3 / 23
by Dr. Lucy D'Agostino McGowan
Types of statistical models
| response | predictor(s) | model |
|---|---|---|
| quantitative | one quantitative | simple linear regression |
| quantitative | two or more (of either kind) | multiple linear regression |
| binary | one (of either kind) | simple logistic regression |
| binary | two or more (of either kind) | multiple logistic regression |
4 / 23
by Dr. Lucy D'Agostino McGowan
Types of statistical models
| variables | predictor | ordinary regression | logistic regression |
|---|---|---|---|
| one: \(x\) | \(\beta_0 + \beta_1 x\) | Response \(y\) | \(\textrm{logit}(\pi)=\log\left(\frac{\pi}{1-\pi}\right)\) |
| several: \(x_1,x_2,\dots,x_k\) | \(\beta_0 + \beta_1x_1 + \dots+\beta_kx_k\) | Response \(y\) | \(\textrm{logit}(\pi)=\log\left(\frac{\pi}{1-\pi}\right)\) |
5 / 23
by Dr. Lucy D'Agostino McGowan
Multiple logistic regression
- ✌️ forms
| Form | Model |
|---|---|
| Logit form | \(\log\left(\frac{\pi}{1-\pi}\right) = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots \beta_kx_k\) |
| Probability form | \(\Large\pi = \frac{e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + \dots \beta_kx_k}}{1+e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + \dots \beta_kx_k}}\) |
6 / 23
by Dr. Lucy D'Agostino McGowan
Steps for modeling
7 / 23
by Dr. Lucy D'Agostino McGowan
Fit
data(MedGPA)glm(Acceptance ~ MCAT + GPA, data = MedGPA, family = "binomial") %>% tidy(conf.int = TRUE)## # A tibble: 3 x 7## term estimate std.error statistic p.value conf.low conf.high## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 (Intercept) -22.4 6.45 -3.47 0.000527 -36.9 -11.2 ## 2 MCAT 0.165 0.103 1.59 0.111 -0.0260 0.383## 3 GPA 4.68 1.64 2.85 0.00439 1.74 8.278 / 23
by Dr. Lucy D'Agostino McGowan
Fit
What does this do?
glm(Acceptance ~ MCAT + GPA, data = MedGPA, family = "binomial") %>% tidy(conf.int = TRUE)## # A tibble: 3 x 7## term estimate std.error statistic p.value conf.low conf.high## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 (Intercept) -22.4 6.45 -3.47 0.000527 -36.9 -11.2 ## 2 MCAT 0.165 0.103 1.59 0.111 -0.0260 0.383## 3 GPA 4.68 1.64 2.85 0.00439 1.74 8.279 / 23
by Dr. Lucy D'Agostino McGowan
Fit
What does this do?
glm(Acceptance ~ MCAT + GPA, data = MedGPA, family = "binomial") %>% tidy(conf.int = TRUE)## # A tibble: 3 x 7## term estimate std.error statistic p.value conf.low conf.high## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 (Intercept) -22.4 6.45 -3.47 0.000527 -36.9 -11.2 ## 2 MCAT 0.165 0.103 1.59 0.111 -0.0260 0.383## 3 GPA 4.68 1.64 2.85 0.00439 1.74 8.2710 / 23
by Dr. Lucy D'Agostino McGowan
Fit
What does this do?
glm(Acceptance ~ MCAT + GPA, data = MedGPA, family = "binomial") %>% tidy(conf.int = TRUE) %>% kable()| term | estimate | std.error | statistic | p.value | conf.low | conf.high |
|---|---|---|---|---|---|---|
| (Intercept) | -22.373 | 6.454 | -3.47 | 0.001 | -36.894 | -11.235 |
| MCAT | 0.165 | 0.103 | 1.59 | 0.111 | -0.026 | 0.383 |
| GPA | 4.676 | 1.642 | 2.85 | 0.004 | 1.739 | 8.272 |
11 / 23
by Dr. Lucy D'Agostino McGowan
Assess
What are the assumptions of multiple logistic regression?
12 / 23
by Dr. Lucy D'Agostino McGowan
Assess
What are the assumptions of multiple logistic regression?
- Linearity
- Independence
- Randomness
12 / 23
by Dr. Lucy D'Agostino McGowan
Assess
How do you determine whether the conditions are met?
- Linearity
- Independence
- Randomness
13 / 23
by Dr. Lucy D'Agostino McGowan
Assess
How do you determine whether the conditions are met?
- Linearity: empirical logit plots
- Independence: look at the data generation process
- Randomness: look at the data generation process (does the spinner model make sense?)
14 / 23
by Dr. Lucy D'Agostino McGowan
Assess
If I have two nested models, how do you think I can determine if the full model is significantly better than the reduced?
15 / 23
by Dr. Lucy D'Agostino McGowan
Assess
If I have two nested models, how do you think I can determine if the full model is significantly better than the reduced?
- We can compare values of \(-2 \log(\mathcal{L})\) (deviance) between the two models
15 / 23
by Dr. Lucy D'Agostino McGowan
Assess
If I have two nested models, how do you think I can determine if the full model is significantly better than the reduced?
- We can compare values of \(-2 \log(\mathcal{L})\) (deviance) between the two models
- Calculate the "drop in deviance" the difference between \((-2 \log(\mathcal{L}_{reduced})) - ( -2 \log(\mathcal{L}_{full}))\)
15 / 23
by Dr. Lucy D'Agostino McGowan
Assess
If I have two nested models, how do you think I can determine if the full model is significantly better than the reduced?
- We can compare values of \(-2 \log(\mathcal{L})\) (deviance) between the two models
- Calculate the "drop in deviance" the difference between \((-2 \log(\mathcal{L}_{reduced})) - ( -2 \log(\mathcal{L}_{full}))\)
- This is a "likelihood ratio test"
15 / 23
by Dr. Lucy D'Agostino McGowan
Assess
If I have two nested models, how do you think I can determine if the full model is significantly better than the reduced?
- We can compare values of \(-2 \log(\mathcal{L})\) (deviance) between the two models
- Calculate the "drop in deviance" the difference between \((-2 \log(\mathcal{L}_{reduced})) - ( -2 \log(\mathcal{L}_{full}))\)
- This is a "likelihood ratio test"
- This is \(\chi^2\) distributed with \(p\) degrees of freedom
15 / 23
by Dr. Lucy D'Agostino McGowan
Assess
If I have two nested models, how do you think I can determine if the full model is significantly better than the reduced?
- We can compare values of \(-2 \log(\mathcal{L})\) (deviance) between the two models
- Calculate the "drop in deviance" the difference between \((-2 \log(\mathcal{L}_{reduced})) - ( -2 \log(\mathcal{L}_{full}))\)
- This is a "likelihood ratio test"
- This is \(\chi^2\) distributed with \(p\) degrees of freedom
- \(p\) is the difference in number of predictors between the full and reduced models
15 / 23
by Dr. Lucy D'Agostino McGowan
Assess
- We want to compare a model with GPA and MCAT to one with only GPA
glm(Acceptance ~ GPA, data = MedGPA, family = binomial) %>% glance()## # A tibble: 1 x 7## null.deviance df.null logLik AIC BIC deviance df.residual## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>## 1 75.8 54 -28.4 60.8 64.9 56.8 53glm(Acceptance ~ GPA + MCAT, data = MedGPA, family = binomial) %>% glance()## # A tibble: 1 x 7## null.deviance df.null logLik AIC BIC deviance df.residual## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>## 1 75.8 54 -27.0 60.0 66.0 54.0 5256.83901 - 54.01419## [1] 2.8216 / 23
by Dr. Lucy D'Agostino McGowan
Assess
- We want to compare a model with GPA and MCAT to one with only GPA
glm(Acceptance ~ GPA, data = MedGPA, family = binomial) %>% glance()## # A tibble: 1 x 7## null.deviance df.null logLik AIC BIC deviance df.residual## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>## 1 75.8 54 -28.4 60.8 64.9 56.8 53glm(Acceptance ~ GPA + MCAT, data = MedGPA, family = binomial) %>% glance()## # A tibble: 1 x 7## null.deviance df.null logLik AIC BIC deviance df.residual## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>## 1 75.8 54 -27.0 60.0 66.0 54.0 52pchisq(56.83901 - 54.01419 , df = 1, lower.tail = FALSE)## [1] 0.092817 / 23
by Dr. Lucy D'Agostino McGowan
Assess
- We want to compare a model with GPA, MCAT, and number of applications to one with only GPA
glm(Acceptance ~ GPA, data = MedGPA, family = "binomial") %>% glance()## # A tibble: 1 x 7## null.deviance df.null logLik AIC BIC deviance df.residual## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>## 1 75.8 54 -28.4 60.8 64.9 56.8 53glm(Acceptance ~ GPA + MCAT + Apps, data = MedGPA, family = "binomial") %>% glance()## # A tibble: 1 x 7## null.deviance df.null logLik AIC BIC deviance df.residual## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <int>## 1 75.8 54 -26.8 61.7 69.7 53.7 51pchisq(56.83901 - 53.68239, df = 2, lower.tail = FALSE)## [1] 0.20618 / 23
by Dr. Lucy D'Agostino McGowan
Use
- We can calculate confidence intervals for the \(\beta\) coefficients: \(\hat\beta\pm z^*\times SE_{\hat\beta}\)
- To determine whether individual explanatory variables are statistically significant, we can calculate p-values based on the \(z\)-statistic of the \(\beta\) coefficients (using the normal distribution)
19 / 23
by Dr. Lucy D'Agostino McGowan
Use
How do you interpret these \(\beta\) coefficients?
glm(Acceptance ~ MCAT + GPA, data = MedGPA, family = "binomial") %>% tidy(conf.int = TRUE)## # A tibble: 3 x 7## term estimate std.error statistic p.value conf.low conf.high## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 (Intercept) -22.4 6.45 -3.47 0.000527 -36.9 -11.2 ## 2 MCAT 0.165 0.103 1.59 0.111 -0.0260 0.383## 3 GPA 4.68 1.64 2.85 0.00439 1.74 8.2720 / 23
by Dr. Lucy D'Agostino McGowan
\(\hat\beta\) interpretation in multiple logistic regression
The coefficient for \(x\) is \(\hat\beta\) (95% CI: \(LB_\hat\beta, UB_\hat\beta\)). A one-unit increase in \(x\) yields a \(\hat\beta\) expected change in the log odds of y, holding all other variables constant.
21 / 23
by Dr. Lucy D'Agostino McGowan
\(e^{\hat\beta}\) interpretation in multiple logistic regression
The odds ratio for \(x\) is \(e^{\hat\beta}\) (95% CI: \(e^{LB_\hat\beta}, e^{UB_\hat\beta}\)). A one-unit increase in \(x\) yields a \(e^{\hat\beta}\)-fold expected change in the odds of y, holding all other variables constant.
22 / 23
by Dr. Lucy D'Agostino McGowan
Summary
| Ordinary regression | Logistic regression | ||
|---|---|---|---|
| test or interval for \(\beta\) | \(t = \frac{\hat\beta}{SE_{\hat\beta}}\) | \(z = \frac{\hat\beta}{SE_{\hat\beta}}\) | |
| t-distribution | z-distribution | ||
| test for nested models | \(F = \frac{\Delta SSModel / p}{SSE_{full} / (n - k - 1)}\) | G = \(\Delta(-2\log\mathcal{L})\) | |
| F-distribution | \(\chi^2\)-distribution |
23 / 23