Back to ML Foundations: The Mental Model

Train/Validation/Test — The Three-Split Discipline

The single piece of ML discipline that separates honest models from theatre. FIND_VIDEO: search 'train validation test split machine learning' — recommended channel: StatQuest / Andrew Ng. Aim for 10 min or under.

4 minutesVideo LessonPDF notes

Watch Train/Validation/Test — The Three-Split Discipline on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Model Prediction Goal — The goal is to build a model that predicts car prices based on historical features like make, age, and mileage.
Train/Test Data Split — The data must be split, typically 80% for training the model and 20% set aside for testing.
Need for Unseen Data — Testing on unseen data is necessary to ensure the model generalizes predictions rather than just memorizing the training set.
Testing Procedure — The model predicts values for the test set features without being shown the actual historical prices.
Evaluation and Comparison — Predicted values are compared against the known actual values in the test set to evaluate performance on unseen data.
Model Approval — If the model performs well on the unseen test data, it is approved because it demonstrates good generalization.

PDF notes

Bottom line: You will learn the fundamental discipline of splitting historical data into training and testing sets for machine learning model development. This process ensures that your model is evaluated on data it has never seen, confirming its ability to generalize predictions accurately.

This lesson on Train/Validation/Test — The Three-Split Discipline is hands-on and example-driven. You will learn the fundamental discipline of splitting historical data into training and testing sets for machine learning model development. This process ensures that your model is evaluated on data it has never seen, confirming its ability to generalize predictions accurately.

What You'll Be Able To Do

Define the purpose of training data in model development.
Explain the necessity of setting aside a test set before training begins.
Calculate a typical 80/20 split ratio for data partitioning.
Evaluate a model's performance using unseen data (the test set).
Distinguish between a model that has learned specific patterns and one that generalizes well.

Topics Covered in Train/Validation/Test — The Three-Split Discipline

Model Prediction Goal (0:00 - 0:55) — The goal is to build a model that predicts car prices based on historical features like make, age, and mileage.
Train/Test Data Split (0:55 - 1:35) — The data must be split, typically 80% for training the model and 20% set aside for testing.
Need for Unseen Data (1:35 - 2:10) — Testing on unseen data is necessary to ensure the model generalizes predictions rather than just memorizing the training set.
Testing Procedure (2:10 - 2:55) — The model predicts values for the test set features without being shown the actual historical prices.
Evaluation and Comparison (2:55 - 3:30) — Predicted values are compared against the known actual values in the test set to evaluate performance on unseen data.
Model Approval (3:30 - 3:50) — If the model performs well on the unseen test data, it is approved because it demonstrates good generalization.

SQL Cheat Sheet

Historical Data — Past observations used to build predictive models, like car sales records
```
SELECT make, model, age, price FROM CARS_DATA;
```
Training Data — The majority of historical data used for the model to learn patterns
```
SELECT * FROM CARS_DATA WHERE split = 'train';
```
Test Data — Unseen data reserved exclusively for evaluating final model performance
```
SELECT make, model, mileage FROM CARS_DATA WHERE split = 'test';
```
80/20 Split — A common rule of thumb for partitioning data into train and test sets
```
SELECT COUNT(*) * 0.8 FROM CARS_DATA;
```
Generalization — The model's ability to accurately predict outcomes on new, unseen data
```
SELECT price_predicted, price_actual FROM CARS_EVALUATION;
```

Comparison Table

Data Set	Typical Size	Role in Modeling
Training Data	~80% of total	Used to adjust model parameters
Test Data	~20% of total	Used to verify generalization ability
Exposure to Model	Fully seen during learning	Never seen during training

Common Pitfalls

Mistake: Evaluating performance only on the training data used to build the model. Avoid: Always compare predicted values against actual values in the unseen test set.
Mistake: Using the test set to adjust model hyperparameters during development. Avoid: Only use the test set once for final, unbiased evaluation.
Mistake: Failing to separate the data before starting any model training process. Avoid: Partition the historical data into 80/20 sets immediately upon collection.

FAQs

Why is the 80/20 split ratio used? It is a common rule of thumb, but the exact percentage depends on the model type, data size, and specific needs. The key is ensuring the test set is large enough to be representative.
What does it mean if the model performs well on training data but poorly on test data? This indicates the model has overfit; it learned the specific patterns of the training set but cannot generalize well to new, unseen data.
If the model performs poorly on the test set, what should I do? You should adjust the model, change the approach, or tweak the hyperparameters. You must then re-evaluate the new model using a fresh, unseen test set.

Common pitfalls

Mistake: Evaluating performance only on the training data used to build the model. Avoid: Always compare predicted values against actual values in the unseen test set.
Mistake: Using the test set to adjust model hyperparameters during development. Avoid: Only use the test set once for final, unbiased evaluation.
Mistake: Failing to separate the data before starting any model training process. Avoid: Partition the historical data into 80/20 sets immediately upon collection.

Hello and welcome back. Today we're talking about training data and test data. You might encounter a question about this on the exam, so it's important to understand these concepts. So, uh imagine you want to build a machine learning model that will predict uh prices of secondhand uh cars. And for each car, you have information about it such as the make and model, the age of the car, the mileage, and the condition of the car. Now, you want to build a machine learning model that you'd put that information into this model and from that it would predict the price of this car. Now, how are you going to train this model? Well, you're going to go and collect lots of data. So let's say you've collected data on hundreds and hundreds of cars, and for each car you know the make and model, the age, the mileage, and the condition, and also you know the price for which that car has sold. So this is historical data. You've collected it. Now you're going to use it to train your model. So, what the approach of training data versus test data dictates is that you should take uh roughly 20% of that data and set it aside, and only use 80% of your data for training. And the exact percentage percentages differ from based on the model, the approach, the needs, your data and so on, but roughly a ballpark, a rule of thumb is taking 80% of your data and using that for training. So you put that through the model, uh you let the model train on that data, learn from that data, and eventually you get a trained model. But now how do you evaluate if this model is really good at predicting or it's just learned uh the specific patterns of that uh of that training data, of that historical data that you showed the model? Maybe it's just really good, it adapted really well to that specific training data. It's not going to be good at predicting, you know, new car data. Or, how can you trust this model is actually good at gen- at generalizing its predictions, and not just very good at that specific um training set that you gave it? Well, the good news is now you have the 20%, this uh test set, uh which the model has not seen at all. It did not see this data during training. So, what you can do is you, well, what you do do is you take uh this um data, and you just take the car information. So, you take the make model, age, mileage, condition, and you separate out the price. You don't include the price. You don't show the model the price. You put those cars, or that information about the cars, into the model and you ask it to predict, "Hey, model, what do you think the prices for these cars should be according to you?" And the model predicts values for the prices. But the good news is because this is your historical data, you actually have the actual value. So you can compare the predicted values to the actual values. And from that, you can evaluate how your model is doing with unseen data. With data it has not previously seen, is it performing uh as good as you would want it to or not? If it's not performing well, then you can adjust it, change it, tweak the hyper parameters, and we'll talk more about those things later. Um, but if you're happy with the result on that, then you can approve it because now you know that uh it's been trained on uh certain historical data, and then it has been tested on data it did not see during training, and it did well even on that data, meaning that it generalizes its predictions well. So, that's the whole concept about training and test data, and that's what it's needed for. Here's a quick summary of what we discussed. I'll look forward to seeing you back here next time, and until then, enjoy the code.

Frequently asked questions

Why is the 80/20 split ratio used?

It is a common rule of thumb, but the exact percentage depends on the model type, data size, and specific needs. The key is ensuring the test set is large enough to be representative.

What does it mean if the model performs well on training data but poorly on test data?

This indicates the model has overfit; it learned the specific patterns of the training set but cannot generalize well to new, unseen data.

If the model performs poorly on the test set, what should I do?

You should adjust the model, change the approach, or tweak the hyperparameters. You must then re-evaluate the new model using a fresh, unseen test set.