Sailing on a boat, headed toward StatQuest.
Join me on this boat, let's go to StatQuest.
It's super cool.
Hello, and welcome to StatQuest. StatQuest is brought to you by the friendly folks in the genetics department at the University of North Carolina at Chapel Hill. Today we're going to be talking about linear regression, aka general linear models part one. There's a lot of parts to linear models, but it's a really cool and powerful concept. So let's get right down to it.
I promise you, I have lots and lots of slides that talk about all the gritty details behind linear regression. But first, let's talk about the main ideas behind it.
The first thing you do in linear regression is use least squares to fit a line to the data.
The second thing you do is calculate r squared.
Lastly, calculate a p-value for r squared.
There are lots of other little things that come up along the way, but these are the three most important concepts behind linear regression.
In the StatQuest "Fitting a Line to Data", we talked about fitting a line to data. Duh.
But let's do a quick review. I'm going to introduce some new terminology in this part of the video, so it's worth watching even if you've already seen the earlier StatQuest. That said, if you need more details, check that StatQuest out.
For this review, we're going to be talking about a data set where we took a bunch of mice, and we measured their size, and we measured their weight.
Our goal is to use mouse weight as a way to predict mouse size.
First, draw a line through the data.
Second, measure the distance from the line to the data, square each distance, and then add them up. Terminology alert: The distance from the line to the data point is called a residual.
Third, rotate the line a little bit.
With the new line, measure the residuals, square them, and then sum up the squares.
Now rotate the line a little bit more.
Sum up the squared residuals, et cetera, et cetera, et cetera. We rotate, and then sum up the squared residuals. Rotate, then sum up the squared residuals. Just keep doing that.
After a bunch of rotations, you can plot the sum of squared residuals and corresponding rotation.
So, in this graph, we have the sum of squared residuals on the y-axis, and the different rotations on the x-axis.
Lastly, you find the rotation that has the least sum of squares.
More details about how this is actually done in practice are provided in the StatQuest on fitting a line to data.
So, we see that this rotation is the one with the least squares. So it will be the one to fit to the data.
This is our least squares rotation superimposed on the original data. Bam! Now we know why the method for fitting a line is called least squares.
Now we have fit a line to the data. This is awesome.
Here's the equation for the line.
Least squares estimated two parameters.
A y-axis intercept, and a slope.
Since the slope is not zero, it means that knowing a mouse's weight will help us make a guess about that mouse's size.
How good is that guess?
Calculating r squared is the first step in determining how good that guess will be.
The StatQuest "R Squared Explained" talks about, you got it, r squared.
Let's do a quick review. I'm also going to introduce some additional terminology, so it's worth watching this part of the video even if you've seen the original StatQuest on r squared.
First, calculate the average mouse size.
Okay, I've just shifted all the data points to the y-axis to emphasize that at this point, we are only interested in mouse size.
Here, I've drawn a black line to show the average mouse size. Bam!
Now, sum the squared residuals.
Just like in least squares, we measure the distance from the mean to the data point, and square it, and then add those squares together.
Terminology alert: We'll call this SS mean for sum of squares around the mean.
Note: The sum of squares around the mean equals the data minus the mean squared.
The variation around the mean equals the data minus the mean squared divided by n.
N is the sample size. In this case, n equals 9.
The shorthand notation is the variation around the mean equals the sum of squares around the mean divided by n, the sample size.
Another way to think about variance is as the average sum of squares per mouse.
Now go back to the original plot and sum up the squared residuals around our least squares fit.
We'll call this SS fit for the sum of squares around the least squares fit.
The sum of squares around the least squares fit is the sum of the distances between the data and the line squared.
Just like with the mean, the variance around the fit is the distance between the line and the data squared divided by n, the sample size.
The shorthand is the variation around the fitted line equals the sum of squares around the fitted line divided by n, the sample size.
Again, we can think of the variation around the fit as the average of the sum of squares around the fit for each mouse.
In general, the variance of something equals the sum of squares divided by the number of those things.
In other words, it's an average of sum of squares. I mention this because it's going to come in handy in a little bit, so keep it in the back of your mind.
Okay, let's step back a little bit. This is the raw variation in mouse size.
And this is the variation around the least squares line.
There's less variation around the line that we fit by least squares. That is to say, the residuals are smaller.
As a result, we say that some of the variation in mouse size is explained by taking mouse weight into account.
In other words, heavier mice are bigger. Lighter mice are smaller.
R squared tells us how much of the variation in mouse size can be explained by taking mouse weight into account.
This is the formula for r squared. It's the variation around the mean minus the variation around the fit divided by the variation around the mean.
Let's look at an example. In this example, the variation around the mean equals 11.1, and the variation around the fit equals 4.4. So we plug those numbers into the equation. The result is that r squared equals 0.6, which is the same thing as saying 60%.
This means there is a 60% reduction in the variance when we take the mouse weight into account.
Alternatively, we can say that mouse weight explains 60% of the variation in mouse size.
We can also use the sum of squares to make the same calculation. This is because when we're talking about variation, everything is divided by n, the sample size. Since everything is scaled by n, we can pull that term out and just use the raw sum of squares.
In this case, the sum of squares around the mean equals 100, and the sum of squares around the fit equals 40. Plugging those numbers into the equation gives us the same value we had before: r squared equals 0.6, which equals 60%.
60% of the sums of squares of the mouse size can be explained by mouse weight.
Here's another example. We're also going to go back to using variation in the calculation, since that's more common.
In this case, knowing mouse weight means you can make a perfect prediction of mouse size.
The variation around the mean is the same as it was before, 11.1. But now the variation around the fitted line equals zero, because there are no residuals.
Plugging the numbers in gives us an r squared equal to 1, which equals 100%.
In this case, mouse weight explains 100% of the variation in mouse size.
Okay, one last example.
In this case, knowing mouse weight doesn't help us predict mouse size.
If someone tells us they have a heavy mouse, well that mouse could either be small or large with equal probability. Similarly, if someone said they had a light mouse, well again, we wouldn't know if it was a big mouse or a small mouse because each of those options is equally likely.
Just like the other two examples, the variation around the mean is equal 11.1. However, in this case, the variation around the fit is also equal 11.1. So we plug those numbers in and we get r squared equals 0, which equals 0%.
In this case, mouse weight doesn't explain any of the variation around the mean.
When calculating the sum of squares around the mean, we collapse the points onto the y-axis just to emphasize the fact that we were ignoring mouse weight.
But we could just as easily draw a line, y equals the mean mouse size, and calculate the sum of squares around the mean around that.
In this example, we applied r squared to a simple equation for a line: y equals 0.1 plus 0.78 times x. This gave us an r squared of 60%, meaning 60% of the variation in mouse size could be explained by mouse weight.
But the concept applies to any equation, no matter how complicated.
First, you measure, square, and sum the distance from the data to the mean.
Then measure, square, and sum the distance from the data to the complicated equation.
Once you've got those two sums of squares, just plug them in and you've got r squared.
Let's look at a slightly more complicated example.
Imagine we wanted to know if mouse weight and tail length did a good job predicting the length of the mouse's body.
So we measure a bunch of mice.
To plot this data, we need a three-dimensional graph.
We want to know how well weight and tail length predict body length. The first mouse we measured had weight equals 2.1, tail length equals 1.3, and body length equals 2.5.
So that's how we plot this data on this 3D graph.
Here's all the data in the graph. The larger circles are points that are closer to us and represent mice that have shorter tails. The smaller circles are points that are further from us and represent mice with longer tails.
Now we do a least squares fit. Since we have the extra term in the equation, representing an extra dimension, we fit a plane instead of a line.
Here's the equation for the plane.
The y value represents body length.
Least squares estimates three different parameters.
The first is the y-intercept. That's when both tail length and mouse weight are equal to zero.
The second parameter, 0.7, is for the mouse weight.
The last term, 0.5, is for the tail length.
If we know a mouse's weight and tail length, we can use the equation to guess the body length.
For example, given the weight and tail length for this mouse, the equation predicts this body length.
Just like before, we can measure the residuals, square them, and then add them up to calculate r squared.
Now, if the tail length, or the z-axis, is useless and doesn't make the sum of squares fit any smaller, then least squares will ignore it by making that parameter equal to zero.
In this case, plugging the tail length into the equation would have no effect on predicting the mouse size.
This means equations with more parameters will never make the sum of squares around the fit worse than equations with fewer parameters.
In other words, this equation: mouse size equals 0.3 plus mouse weight plus flip of a coin plus favorite color plus astrological sign plus extra stuff, will never perform worse than this equation: mouse size equals 0.3 plus mouse weight.
This is because least squares will cause any term that makes sum of squares around the fit worse to be multiplied by zero, and in a sense, no longer exist.
Now, due to random chance, there is a small probability that the small mice in the data set might get heads more frequently than large mice.
If this happened, then we'd get a smaller sum of squares fit and a better r squared.
Womp womp. Here's the frowny face of sad times.
The more silly parameters we add to the equation, the more opportunities we have for random events to reduce sum of squares fit and result in a better r squared.
Thus, people report an adjusted r squared value that, in essence, scales r squared by the number of parameters.
R squared is awesome, but it's missing something.
What if all we had were two measurements?
We'd calculate the sum of squares around the mean. In this case, that would be 10.
Then we'd calculate the sum of squares around the fit, which equals 0.
The sum of squares around the fit equals 0 because you can always draw a straight line to connect any two points.
What this means is when we calculate r squared by plugging the numbers in, we're going to get 100%.
100% is a great number. We've explained all the variation. But any two random points will give us the exact same thing. It doesn't actually mean anything.
We need a way to determine if the r squared value is statistically significant.
We need a p-value.
Before we calculate the p-value, let's review the main concepts behind r squared one last time.
The general equation for r squared is the variance around the mean minus the variance around the fit divided by the variance around the mean.
In our example, this means the variation in the mouse size minus the variation after taking weight into account divided by the variation in mouse size.
In other words, r squared equals the variation in mouse size explained by weight divided by the variation in mouse size without taking weight into account.
In this particular example, r squared equals 0.6, meaning we saw a 60% reduction in variation once we took mouse weight into account.
Now that we have a thorough understanding of the ideas behind r squared, let's talk about the main ideas behind calculating a p-value for it.
The p-value for r squared comes from something called F.
F is equal to the variation in mouse size explained by weight divided by the variation in mouse size not explained by weight.
The numerators for r squared and for F are the same.
That is to say, it's the reduction in variance when we take the weight into account.
The denominator is a little different.
These dotted lines, the residuals, represent the variation that remains after fitting the line. This is the variation that is not explained by weight.
So together, we have the variation in mouse size explained by weight divided by the variation in mouse size not explained by weight.
Now let's look at the underlying mathematics. Just as a reminder, here's the equation for r squared.
This is the general equation that will tell us if r squared is significant.
The meat of these two equations are very similar and rely on the same sums of squares.
Like we said before, the numerators are the same.
In our mouse size and weight example, the numerator is the variation in mouse size explained by weight.
And the sum of squares around the fit is just the residuals squared and summed up around the fitted line. So that's the variation that the fit does not explain.
These numbers over here are the degrees of freedom.
They turn the sums of squares into variances.
I'm going to dedicate a whole StatQuest to degrees of freedom, but for now, let's see if we can get an intuitive feel for what they're doing here.
Let's start with these.
P fit is the number of parameters in the fit line.
Here's the equation for the fit line in a general format. We just have the y-intercept plus the slope times x. The y-intercept and the slope are two separate parameters.
That means P fit equals 2.
P mean is the number of parameters in the mean line.
In general, that equation is y equals the y-intercept. That's what gives us a horizontal line that cuts through the data.
In this case, the y-intercept is the mean value.
This equation just has one parameter.
Thus, P mean equals 1.
Both equations have a parameter for the y-intercept.
However, the fit line has one extra parameter, the slope. In our example, this slope is the relationship between weight and size.
In this example, P fit minus P mean equals 2 minus 1, which equals 1.
The fit has one extra parameter, mouse weight.
Thus, the numerator is the variance explained by the extra parameter. In our example, that's the variance in mouse size explained by mouse weight.
If we had used mouse weight and tail length to explain variation in size, then we would end up with an equation that had three parameters, and P fit would equal 3.
Thus, P fit minus P mean would equal 3 minus 1, which equals 2.
Now the fit has two extra parameters, mouse weight and tail length.
With the fancier equation for the fit, the numerator is the variance in mouse size explained by mouse weight and tail length.
Now let's talk about the denominator for our equation for F.
The denominator is the variation in mouse size not explained by the fit.
That is to say, it's the sum of squares of the residuals that remain after we fit our new line to the data.
Why divide sum of squares fit by n minus P fit instead of just n?
Intuitively, the more parameters you have in your equation, the more data you need to estimate them. For example, you only need two points to estimate a line, but you need three points to estimate a plane.
If the fit is good, then the variation explained by the extra parameters in the fit will be a large number, and the variation not explained by the extra parameters in the fit will be a small number. That makes F a really large number.
Now, that question we've all been dying to know the answer to, how do we turn this number into a p-value?
Conceptually, generate a set of random data.
Calculate the mean and the sum of squares around the mean.
Calculate the fit and the sum of squares around the fit.
Now plug all those values into our equation for F, and that will give us a number. In this case, that number is 2.
Now plot that number in a histogram.
Now generate another set of random data.
Calculate the mean and the sum of squares around the mean.
Then calculate the fit and the sum of squares around the fit.
Plug those values into our equation for F, and in this case, we get F equals 3. So we then plug that value into our histogram.
And then we repeat with yet another set of random data. In this case, we got F equals 1. That's plotted on our histogram.
And we just keep generating more and more random data sets, calculating the sums of squares, plugging them into our equation for F, and plotting the results in our histogram.
Now, imagine we did that hundreds, if not millions of times.
When we're all done with our random data sets, we return to our original data set.
We then plug the numbers into our equation for F. In this case, we got F equals 6.
The p-value is the number of more extreme values divided by all of the values.
So in this case, we have the value at F equals 6 and the value at F equals 7 divided by all the other randomizations that we created originally. If this concept is confusing to you, I have a StatQuest that explains p-values. So check that one out.
Bam!
You can approximate the histogram with a line. In practice, rather than generating tons of random data sets, people use the line to calculate the p-value.
Here's an example of one standard F distribution that people use to calculate p-values. The degrees of freedom determine the shape.
The red line represents another standard F distribution that people use to calculate p-values. In this case, the sample size used to draw the red line is smaller than the sample size used to draw the blue line.
Notice that when n minus P fit equals 10, the distribution tapers off faster.
This means that the p-value will be smaller when there are more samples relative to the number of parameters in the fit equation.
Triple bam!
Hooray! We finally got our p-value. Now let's review the main ideas.
Given some data that you think are related, linear regression quantifies the relationship in the data. This is r squared. This needs to be large.
It also determines how reliable that relationship is. This is the p-value that we calculated with F. This needs to be small.
You need both to have an interesting result.
Hooray! We've made it to the end of another exciting StatQuest. Wow, this was a long one. I hope you had a good time. If you like this and want to see more StatQuests like it, why not subscribe to my channel? It's real easy, just click the red button. And, if you have any ideas of StatQuests that you'd like me to create, just put them in the comments below. That's all there is to it. All right, tune in next time for another really exciting StatQuest.
