Back to ML Foundations: The Mental Model

The Bias-Variance Tradeoff

Why models can be 'too simple' or 'too complex' — and what to do when you suspect one. FIND_VIDEO: search 'bias variance tradeoff explained' — recommended channel: StatQuest. Aim for 10 min or under.

17 minutesVideo LessonPDF notes

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Core Principle — The Bias-Variance Tradeoff is a fundamental principle applicable to every machine learning model.
Restricted Environment Analogy — The story of a child raised in a highly restricted environment illustrates the concept of a model that cannot handle real-world complexity.
Real Data Uncertainty — Aiming for 100% accuracy on training data fails because the training set is only a small, unquantifiable fraction of the total real-world data.
Data Splitting — The dataset is separated into blue dots for training and pink dots for testing the model.
Linear Regression Fit — Linear regression fits the training data by minimizing the sum of squares, achieving an initial fit of roughly 80%.
Polynomial Overfitting — A complex polynomial function is built to achieve a high training accuracy of 98% by fitting the training points closely.
Testing Overfit Model — The 98% accurate polynomial model performs poorly, achieving only 55-65% accuracy on the unseen testing data.
Testing Linear Model — The simpler linear model maintains a consistent performance of 70-75% on the testing data, demonstrating better generalization.
Defining Bias and Variance — Bias is defined as the distance between the mean of the values and the line, while variance measures the spread of the data.
Tradeoff Necessity — If high training accuracy results in poor testing performance, a bias-variance tradeoff or relaxation is necessary.

PDF notes

Bottom line: You will learn why models should not aim for 100% accuracy on training data and how to balance model complexity to ensure reliable predictions on unseen (testing) data. You will be able to identify when a model suffers from high bias (underfitting) or high variance (overfitting).

This lesson on The Bias-Variance Tradeoff is hands-on and example-driven. You will learn why models should not aim for 100% accuracy on training data and how to balance model complexity to ensure reliable predictions on unseen (testing) data. You will be able to identify when a model suffers from high bias (underfitting) or high variance (overfitting).

What You'll Be Able To Do

Define the Bias-Variance Tradeoff principle in machine learning.
Explain why a model built with 100% training accuracy often fails in the real world.
Differentiate between model performance metrics on training data versus testing data.
Interpret the concepts of bias and variance in the context of model fitting.
Identify the need for 'relaxation' in model fitting to improve generalization.

Topics Covered in The Bias-Variance Tradeoff

Core Principle (0:00 - 0:30) — The Bias-Variance Tradeoff is a fundamental principle applicable to every machine learning model.
Restricted Environment Analogy (0:30 - 2:30) — The story of a child raised in a highly restricted environment illustrates the concept of a model that cannot handle real-world complexity.
Real Data Uncertainty (2:30 - 4:30) — Aiming for 100% accuracy on training data fails because the training set is only a small, unquantifiable fraction of the total real-world data.
Data Splitting (4:30 - 6:30) — The dataset is separated into blue dots for training and pink dots for testing the model.
Linear Regression Fit (6:30 - 8:30) — Linear regression fits the training data by minimizing the sum of squares, achieving an initial fit of roughly 80%.
Polynomial Overfitting (8:30 - 10:30) — A complex polynomial function is built to achieve a high training accuracy of 98% by fitting the training points closely.
Testing Overfit Model (10:30 - 12:30) — The 98% accurate polynomial model performs poorly, achieving only 55-65% accuracy on the unseen testing data.
Testing Linear Model (12:30 - 14:30) — The simpler linear model maintains a consistent performance of 70-75% on the testing data, demonstrating better generalization.
Defining Bias and Variance (14:30 - 16:00) — Bias is defined as the distance between the mean of the values and the line, while variance measures the spread of the data.
Tradeoff Necessity (16:00 - 17:00) — If high training accuracy results in poor testing performance, a bias-variance tradeoff or relaxation is necessary.

SQL Cheat Sheet

Bias-Variance Tradeoff — Balancing model complexity for generalization to real-world data
Bias — Distance between the mean prediction and the true mean value
```
SELECT AVG(income) FROM data;
```
Variance — Measure of how spread out the data points or predictions are
```
SELECT STDDEV(income) FROM data;
```
Training Data — Data subset used to fit the model parameters and structure
```
SELECT * FROM data WHERE split = 'train';
```
Testing Data — Unseen data subset used to evaluate model generalization ability
```
SELECT * FROM data WHERE split = 'test';
```
Overfitting — Model fits training data perfectly but fails on testing data

Comparison Table

Model Type	Training Accuracy	Testing Accuracy (Generalization)
Linear Regression (Simple)	~80% (Relaxed fit)	~70-75% (Consistent)
Polynomial Function (Complex)	~98% (Overfit)	~55-65% (Fails miserably)
High Bias (Underfitting)	Low	Low

Common Pitfalls

Mistake: Assuming training data represents all possible real data. Avoid: Always reserve a portion for testing performance.
Mistake: Aiming for 100% accuracy on the training set. Avoid: Accept a relaxed fit to allow for real-world variance.
Mistake: Judging model success based only on training accuracy. Avoid: Judge model success solely on testing data performance.

FAQs

Why is 100% training accuracy bad? It indicates the model has memorized the noise and specific patterns of the training set, which results in poor performance on new, unseen data (overfitting).
What does 'relaxing' the fit mean? It means accepting a slightly lower accuracy on the training data to build a simpler model that generalizes better to the unknown real data.
How is Bias defined mathematically? Bias is the distance between the mean of all predicted values and the mean of the true values in the dataset.
What is the relationship between training data and real data? Training data is only an X percent sample of the infinite, unknown real data, which is why models must generalize beyond the sample.

Common pitfalls

Mistake: Assuming training data represents all possible real data. Avoid: Always reserve a portion for testing performance.
Mistake: Aiming for 100% accuracy on the training set. Avoid: Accept a relaxed fit to allow for real-world variance.
Mistake: Judging model success based only on training accuracy. Avoid: Judge model success solely on testing data performance.

Okay, bias variance trade off. Bias variance trade off is a fundamental method which applies to any and every model that you will build. It doesn't it doesn't mean that you have to see this see this in the light of linear regression or polynomial or SVM. It is something common to the principle of machines learning. Okay, so that is bias and variance trade off. So I'm going to start off with this one with a story, sort of a supporting story for it because yesterday I had already done it but you guys are a different zone. So let's just clearly understand what it is today. I'm going to start off with a story. I'm going to start off with this uh interesting um thing that happens with every one of us. So we've all we all got, you know, our parents and guardians and mentors. Okay, mentors. Or gurus, whoever it is, right? We all have all of this, right? Parents, guardians, mentors, and gurus. And the all of these play a very important role in shaping up the mind of their student or children. Now, if a parent keeps the student or a child or a student in a very restricted environment, restricted. Follow the idea. Restricted environment, with lots of rules. the child grows. So the child grows in a very controlled environment thinking that that is all life is. very very restricted environment. Tell me guys, what happens to this child when he enters real life? Real life contains social gatherings, and real life contains living in hostels, sharing rooms with others who later on become friends, but if he doesn't agree to share, then this will never happen. Right? Correct? I want your, you know, involvement with yes or no. Yes, sir. Yes, yes. So is it good to bring up a child in extremely restricted environment? Or would you rather have it and liberal environment? Which one would you choose? One or two? Liberal, sir. Two. Two, right? Liberal. So this is the same thing as that is the trade-off, what we talk about in machine learning. When we say we need a bias and variance trade-off, we are trying to build the model not with 100% accuracy on the training data. This is what if you build with 100% accuracy on training data, then it will fail on testing data. Why it is so? It is so because your training data is you don't even know whether it is how many how much percent of the real data. It's certain X percent of the real data. Real data is where all of the world and in all of the in future whatever data gets generated, that is the real data. So there is no you don't know how much it is. That's why we don't know how much it is. That's why whenever some people give us a lot of data saying that train the model on this, you don't exactly know how much of this is actually that. Whether it is X percent, 10 percent, 50, you don't know. Because we don't know how much real data is there. Real data might have its own large variance and its own large bias. That's why when you train the model, you should be you should relax. Relax on how much variance and bias the model is capturing. Is capturing on what? On on training data. If you relax here, you are taking into consideration the fact that you don't know exactly how many different kinds of variance and biases occur in the real data. That is why we need bias and variance trade-off. Anybody had any doubts with this figure, this diagram? No, sir. This one was clear. This was just a pictorial representation. Anyone? I'll move. No. So yesterday I had a pictorial representation of a of data. Of data. What kind of data? Age and income. I request everyone to mute their microphones when not in use. So that we don't have the echo back. Okay. So we have income and age. So we've plotted it. Age and income we've plotted it. And uh this is the whole data. Okay? I have now selected the pink ones for testing and blue dots for the training. So what I'm going to do, I'm going to separate them. Separate them for training and testing. We are going to build the model on training data and we're going to test it on the testing data. See, it's the same thing. I've separated. You can correlate it. It's the same thing. I've just put the points. I've taken the dots from here and I've put them separately here. Okay? Nothing nothing much I've done. It is very simply I've just selected and put it there. Now, we've done linear regression many times. Let's apply linear regression on it. on our training data. So what's happening in linear regression? We have this training data, the blue points and we are trying to fit a model. And that model because of least square methods, it will change and finally settle where? Settle with the with what value it will settle? It will settle where the sums of the sum of squares of the distance between the point and the line is minimized. That's where it settles, correct? So once it settles, we've got this model. And roughly you can see the model fit. the R-squared value which explains or which gives us the idea of the model fit. It's come up to be 80% here. Correct? This one, 80%. You can just see, you know, I'm just saying I'm not considering this point to be a part of it, and maybe there was one more point here. this point to be a part of that. So that's what I have mentioned as That's why I've taken 80% that it is a 80% fit. So 80% is not enough. Let's say we say 80% is not enough. We need better. So I took the same data, the same data, plotted it back again, and we built a polynomial function on it. Polynomial gave us 98%. Because why polynomial gives us 98? Because we give lots of order. We give it, take two, three, four, five orders, whatever you require, but build me the function. So on your training data, you achieved a very nice accuracy. But your accuracy is not supposed to be on training data. Your accuracy is supposed to be on your testing data. So the problem is when you are building it, the mistake that we do is that when we build on training data and we keep thinking that if our model is fitting the training data close to 98 and 100%, then we are done. All our next you know, real world data, it's going to give us best prediction. But it turns out that the same model when we applied on testing data, the testing data was here. See the testing data is here. The testing data, right? Same thing. Just look at this. The same thing I have put here and the same figure when I brought it here, if you want to see it again, I brought the same figure here. Or I can take it here also. This is our model. So I put it here. But it is not touching. Here whatever it was, it's 98% it is, you know, making sure that it has crossed all the points. But here it is not so. The same positions where we started. How did I know that I have to put it here? It happens to be exactly where it has to cut the C Y-axis, okay? The C intercept decides exactly where this model comes. Okay? So that's how I put it exactly over here. So with that in mind, when we see the points, now we see many of them are outside the range. So you're roughly about, you had 98%. Even if you don't agree with me for 60, 55, this is roughly about 65% fit. Let's go and check out the testing data performance of our linear regression model. Here there is something called consistency. Okay? Even though it was 80% here, and even though it is 70% or if you if you don't want to agree with 80%, I would say 70 to 75% accuracy is still maintained with this. It is still capturing at least 75% of the variance and roughly 75% of the bias also. The variance is how spread out these are. And the bias is how far away the mean of all this is how far away the mean of all these values from this line. Okay? That is what it is. Bias is you take the mean value of all of this, you get one point. How do you take the mean value of it? Everything is age comma income, no? So it is x comma y. All of this is x comma y. So you take x and y, mean of x, mean of y, you get one point. That point could be somewhere here. So if that point lies here, then the distance between this and this is the bias. And variance is you know, you've calculated variance. You take the standard deviation. What is variance? It's a measure of how spread out the data is, right? Variance is denoted by sigma square. Correct? Yes. So that value is saying how spread out this value is. So when we build a model, don't try to capture the entire variance. Even if you don't capture it, even if you are with 80% or 85%, it's okay. Now when do you come to this point? When do you come to this conclusion? You come to this conclusion when you build the model with 95% accuracy, but it is failing miserably on the training data. That's when you, this is important. Yes, you're yes, you you are right in trying to fit your model properly. It's not a sin, it's not a mistake to you know, have that ambition to train it properly. But when you have achieved 98% on your training data and when you come to testing data, you're not even coming to 90%, neither 85, you're simply floating around 55 and 65, then you should assume that there has to be a bias variance trade-off or relaxation required. That's all there is to this. Okay, so I guess

Frequently asked questions

Why is 100% training accuracy bad?

It indicates the model has memorized the noise and specific patterns of the training set, which results in poor performance on new, unseen data (overfitting).

What does 'relaxing' the fit mean?

It means accepting a slightly lower accuracy on the training data to build a simpler model that generalizes better to the unknown real data.

How is Bias defined mathematically?

Bias is the distance between the mean of all predicted values and the mean of the true values in the dataset.

What is the relationship between training data and real data?

Training data is only an X percent sample of the infinite, unknown real data, which is why models must generalize beyond the sample.