Logistic Regression and the GLM Family

The link between linear regression and binary classification. Plus the GLM framework that generalizes to counts, proportions, durations. FIND_VIDEO: search 'logistic regression GLM tutorial' — recommended channel: StatQuest. Aim for 11 min or under.

9 minutesVideo LessonPDF notes

Watch Logistic Regression and the GLM Family on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Introduction — Logistic Regression is a technique used for both traditional statistics and machine learning.
Linear Regression Review — Linear Regression predicts continuous outcomes, fits a line, and uses R^2 and P-values for evaluation, including in multiple regression.
Comparing Models — Simple linear models can be compared to complicated ones to determine if extra predictors are necessary.
Logistic Regression Basics — Logistic Regression predicts binary outcomes, fits an S-shaped curve from 0 to 1, and the output probability is used for classification.
Predictor Variables — Logistic Regression can use both continuous data like weight and discrete data like genotype.
Variable Significance — Wald's test is used to determine if individual variables significantly contribute to the prediction.
Maximum Likelihood — Logistic Regression uses Maximum Likelihood to fit the curve by finding the parameters that maximize the probability of observing the entire dataset.

PDF notes

Bottom line: You will be able to distinguish between Linear and Logistic Regression based on their output type and fitting methods. You can apply Logistic Regression to predict binary outcomes, interpret the resulting probabilities, and identify useful predictors using statistical tests.

This lesson on Logistic Regression and the GLM Family is hands-on and example-driven. You will be able to distinguish between Linear and Logistic Regression based on their output type and fitting methods. You can apply Logistic Regression to predict binary outcomes, interpret the resulting probabilities, and identify useful predictors using statistical tests.

What You'll Be Able To Do

Differentiate between continuous and binary prediction tasks suitable for Linear vs. Logistic Regression.
Explain the role of the S-shaped logistic function in transforming predictor variables into probabilities.
Identify the primary method (Maximum Likelihood) used to fit the Logistic Regression curve.
Evaluate the usefulness of predictor variables in a logistic model using significance tests like Wald's test.
Describe how a probability output from Logistic Regression is converted into a classification decision.

Topics Covered in Logistic Regression and the GLM Family

Introduction (0:00 - 0:30) — Logistic Regression is a technique used for both traditional statistics and machine learning.
Linear Regression Review (0:30 - 2:40) — Linear Regression predicts continuous outcomes, fits a line, and uses R^2 and P-values for evaluation, including in multiple regression.
Comparing Models (2:40 - 3:40) — Simple linear models can be compared to complicated ones to determine if extra predictors are necessary.
Logistic Regression Basics (3:40 - 5:40) — Logistic Regression predicts binary outcomes, fits an S-shaped curve from 0 to 1, and the output probability is used for classification.
Predictor Variables (5:40 - 6:45) — Logistic Regression can use both continuous data like weight and discrete data like genotype.
Variable Significance (6:45 - 7:45) — Wald's test is used to determine if individual variables significantly contribute to the prediction.
Maximum Likelihood (7:45 - 10:00) — Logistic Regression uses Maximum Likelihood to fit the curve by finding the parameters that maximize the probability of observing the entire dataset.

SQL Cheat Sheet

Linear Regression — Predicts continuous outcomes by fitting a straight line
```
SELECT weight, size FROM mice_data WHERE size > 10;
```
Logistic Regression — Predicts probability of binary outcome using S-curve
```
SELECT weight, genotype, is_obese FROM mice_data;
```

Least Squares — Minimizes sum of squared residuals to fit linear model

SELECT SUM(POWER(actual_size - predicted_size, 2)) FROM predictions;

Maximum Likelihood — Finds curve parameters that maximize probability of observed data

R^2 — Measures proportion of variance explained by the linear model

SELECT CORR(weight, size) * CORR(weight, size) FROM mice_data;

Wald's Test — Determines if a variable's effect is significantly non-zero

SELECT variable, p_value FROM logistic_model_results WHERE p_value < 0.05;

Comparison Table

Feature	Linear Regression	Logistic Regression
Output Type	Continuous (e.g., size)	Probability (0 to 1) / Binary
Function Shape	Straight line	S-shaped logistic curve
Fitting Method	Least Squares	Maximum Likelihood
Evaluation Metric	R^2 and P-value	Wald's Test (for variables)

Common Pitfalls

Mistake: Assuming Logistic Regression uses Least Squares for fitting the S-curve. Avoid: Remember LogR uses Maximum Likelihood because residuals are not defined.
Mistake: Trying to calculate R^2 to compare simple and complex logistic models. Avoid: LogR lacks the residual concept, making R^2 calculation impossible.
Mistake: Thinking LogR only handles continuous predictors like weight. Avoid: LogR handles both continuous (weight) and discrete (genotype) variables.
Mistake: Confusing the probability output with the final classification. Avoid: Probability must be compared to a threshold (usually 50%) for classification.

FAQs

Why can't Logistic Regression use R^2 like Linear Regression? R^2 relies on minimizing and measuring residuals, which are not defined in the context of the S-shaped logistic function.
How is the probability output of LogR used for classification? The probability is compared against a threshold (typically 0.5 or 50%); if higher, the sample is classified as positive.
What is the purpose of Wald's test in Logistic Regression? It determines if an individual predictor variable contributes significantly to the model, meaning its effect is different from zero.
Is Linear Regression considered a form of machine learning? Yes, using data to predict an outcome, even with simple linear models, falls under the category of machine learning.

Common pitfalls

Mistake: Assuming Logistic Regression uses Least Squares for fitting the S-curve. Avoid: Remember LogR uses Maximum Likelihood because residuals are not defined.
Mistake: Trying to calculate R^2 to compare simple and complex logistic models. Avoid: LogR lacks the residual concept, making R^2 calculation impossible.
Mistake: Thinking LogR only handles continuous predictors like weight. Avoid: LogR handles both continuous (weight) and discrete (genotype) variables.
Mistake: Confusing the probability output with the final classification. Avoid: Probability must be compared to a threshold (usually 50%) for classification.

If you can fit a line, you can fit a squiggle. If you can make me laugh, you can make me giggle. StatQuest. Hello, I'm Josh Starmer and welcome to StatQuest. Today we're going to talk about logistic regression. This is a technique that can be used for traditional statistics, as well as machine learning. So let's get right to it. Before we dive into logistic regression, let's take a step back and review linear regression. In another StatQuest, we talked about linear regression. We had some data: weight and size. Then we fit a line to it. And with that line, we could do a lot of things. First, we could calculate $R^2$ and determine if weight and size are correlated. Large values imply a large effect. And second, calculate a $P$ value to determine if the $R^2$ value is statistically significant. And third, we could use the line to predict size given weight. If a new mouse has this weight, then this is the size that we predict from the weight. Although we didn't mention it at the time, using data to predict something falls under the category of machine learning. So plain old linear regression is a form of machine learning. We also talked a little bit about multiple regression. Now we are trying to predict size using weight and blood volume. Alternatively, we could say that we are trying to model size using weight and blood volume. Multiple regression did the same things that normal regression did. We calculated $R^2$ and we calculated the $P$ value. And we could predict size using weight and blood volume. And this makes multiple regression a slightly fancier machine learning method. We also talked about how we can use discrete measurements like genotype to predict size. If you're not familiar with the term genotype, don't freak out. It's no big deal. Just know that it refers to different types of mice. Lastly, we could compare models. So on the left side, we've got normal regression using weight to predict size. And we can compare those predictions to the ones we get from multiple regression, where we're using weight and blood volume to predict size. Comparing the simple model to the complicated one tells us if we need to measure weight and blood volume to accurately predict size, or if we can get away with just weight. Now that we remember all the cool things we can do with linear regression, let's talk about logistic regression. Logistic regression is similar to linear regression except logistic regression predicts whether something is true or false instead of predicting something continuous like size. These mice are obese, and these mice are not. Also, instead of fitting a line to the data, logistic regression fits an S-shaped logistic function. The curve goes from 0 to 1. And that means that the curve tells you the probability that a mouse is obese based on its weight. If we weigh a very heavy mouse, there is a high probability that the new mouse is obese. If we weigh an intermediate mouse, then there is only a 50% chance that the mouse is obese. Lastly, there's only a small probability that a light mouse is obese. Although logistic regression tells the probability that a mouse is obese or not, it's usually used for classification. For example, if the probability a mouse is obese is greater than 50%, then we'll classify it as obese. Otherwise, we'll classify it as not obese. Just like with linear regression, we can make simple models. In this case, we can have obesity predicted by weight, or more complicated models. In this case, obesity is predicted by weight and genotype. In this case, obesity is predicted by weight and genotype and age. And lastly, obesity is predicted by weight, genotype, age, and astrological sign. In other words, just like linear regression, logistic regression can work with continuous data like weight and age, and discrete data like genotype and astrological sign. We can also test to see if each variable is useful for predicting obesity. However, unlike normal regression, we can't easily compare the complicated model to the simple model. And we'll talk more about why in a bit. Instead, we just test to see if a variable's effect on the prediction is significantly different from zero. If not, it means that the variable is not helping the prediction. We use Wald's test to figure this out. We'll talk about that in another StatQuest. In this case, the astrological sign is totes useless. That's statistical jargon for not helping. That means we can save time and space in our study by leaving it out. Logistic regression's ability to provide probabilities and classify new samples using continuous and discrete measurements makes it a popular machine learning method. One big difference between linear regression and logistic regression is how the line is fit to the data. With linear regression, we fit the line using least squares. In other words, we find the line that minimizes the sum of the squares of these residuals. We also use the residuals to calculate $R^2$ and to compare simple models to complicated models. Logistic regression doesn't have the same concept of a residual, so it can't use least squares, and it can't calculate $R^2$. Instead, it uses something called maximum likelihood. There's a whole StatQuest on maximum likelihood, so see that for details, but in a nutshell, you pick a probability scaled by weight of observing an obese mouse, just like this curve. And you use that to calculate the likelihood of observing a non-obese mouse that weighs this much. And then you calculate the likelihood of observing this mouse. And you do that for all of the mice. And lastly, you multiply all of those likelihoods together. That's the likelihood of the data given this line. Then you shift the line and calculate a new likelihood of the data, and then shift the line and calculate the likelihood again, and again. Finally, the curve with the maximum value for the likelihood is selected. Bam! In summary, logistic regression can be used to classify samples, and it can use different types of data, like size and or genotype, to do that classification. And it can also be used to assess what variables are useful for classifying samples, i.e., astrological sign is totes useless. Hooray, we've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you have suggestions for future StatQuests, well, put them in the comments below. Until next time, quest on!

Frequently asked questions

Why can't Logistic Regression use R^2 like Linear Regression?

R^2 relies on minimizing and measuring residuals, which are not defined in the context of the S-shaped logistic function.

How is the probability output of LogR used for classification?

The probability is compared against a threshold (typically 0.5 or 50%); if higher, the sample is classified as positive.

What is the purpose of Wald's test in Logistic Regression?

It determines if an individual predictor variable contributes significantly to the model, meaning its effect is different from zero.

Is Linear Regression considered a form of machine learning?

Yes, using data to predict an outcome, even with simple linear models, falls under the category of machine learning.