Back to ML Foundations: The Mental Model

Cross-Validation — When Data is Scarce

Make every row count. K-fold and its variants for robust evaluation. FIND_VIDEO: search 'cross validation k-fold stratified' — recommended channel: StatQuest. Aim for 10 min or under.

6 minutesVideo LessonPDF notes

Watch Cross-Validation — When Data is Scarce on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Introduction & Goal — The lesson introduces the goal of using patient variables to predict heart disease using machine learning.
Comparing Methods — Cross-validation is introduced as the tool necessary to compare different machine learning methods like Logistic Regression or SVM.
Training vs. Testing — Data must be split to both train (estimate parameters) and test (evaluate performance) the algorithm.
Bad Approaches — Reusing all data or using a fixed, arbitrary split (e.g., 75/25) are shown to be flawed methods for evaluation.
K-Fold Mechanism — Cross-validation solves the splitting problem by systematically rotating which block of data is used for testing.
Summarizing Results — After all blocks have been tested, the overall performance is summarized to select the best performing method.
Types of CV — The number of blocks (K) is arbitrary, leading to K-fold CV (e.g., 10-fold) and Leave-One-Out CV.
Tuning Parameters — Cross-validation can also be used to optimize the value of a tuning parameter that is not estimated by the data.

PDF notes

Bottom line: You will be able to explain why cross-validation is essential for reliably comparing machine learning models when data is limited. You will know the difference between training and testing data and how k-fold cross-validation ensures every data point contributes to testing.

This lesson on Cross-Validation — When Data is Scarce is hands-on and example-driven. You will be able to explain why cross-validation is essential for reliably comparing machine learning models when data is limited. You will know the difference between training and testing data and how k-fold cross-validation ensures every data point contributes to testing.

What You'll Be Able To Do

Define the purpose of training and testing phases in algorithm development.
Compare and contrast different machine learning methods using test data performance.
Explain the mechanism of k-fold cross-validation.
Calculate the appropriate number of training and testing blocks for 4-fold CV.
Identify scenarios where a tuning parameter requires cross-validation for optimization.

Topics Covered in Cross-Validation — When Data is Scarce

Introduction & Goal (0:00 - 0:45) — The lesson introduces the goal of using patient variables to predict heart disease using machine learning.
Comparing Methods (0:45 - 1:20) — Cross-validation is introduced as the tool necessary to compare different machine learning methods like Logistic Regression or SVM.
Training vs. Testing (1:20 - 2:15) — Data must be split to both train (estimate parameters) and test (evaluate performance) the algorithm.
Bad Approaches (2:15 - 3:05) — Reusing all data or using a fixed, arbitrary split (e.g., 75/25) are shown to be flawed methods for evaluation.
K-Fold Mechanism (3:05 - 4:10) — Cross-validation solves the splitting problem by systematically rotating which block of data is used for testing.
Summarizing Results (4:10 - 4:45) — After all blocks have been tested, the overall performance is summarized to select the best performing method.
Types of CV (4:45 - 5:20) — The number of blocks (K) is arbitrary, leading to K-fold CV (e.g., 10-fold) and Leave-One-Out CV.
Tuning Parameters (5:20 - 5:50) — Cross-validation can also be used to optimize the value of a tuning parameter that is not estimated by the data.

SQL Cheat Sheet

Training — Estimate parameters to define the algorithm's shape or curve

SELECT EstimateParameters(chest_pain, blood_circulation) FROM PatientData WHERE is_train = TRUE;

Testing — Evaluate method performance on data it has not seen

SELECT Predict(heart_disease) FROM PatientData WHERE is_test = TRUE;

Cross-Validation — Compares methods by systematically rotating training and testing blocks
```
SELECT AVG(accuracy) FROM CV_Results WHERE method = 'Logistic Regression';
```
K-Fold Cross-Validation — Divides data into K blocks, rotating one block for testing
```
SELECT * FROM PatientData WHERE fold_id = 1; -- Test set for Fold 1
```

Leave-One-Out CV — Extreme case where each individual sample is tested alone

SELECT * FROM PatientData WHERE patient_id = 1; -- Test set of size 1

Tuning Parameter — A parameter value that is guessed rather than estimated by the data
```
SELECT BestValue(tuning_param) FROM CV_Optimization;
```

Comparison Table

Data Split Approach	Training Data Used	Testing Data Used
Terrible Approach	100%	0% (Reused)
Simple Split (75/25)	First 75%	Last 25%
K-Fold Cross-Validation	K-1 blocks (e.g., 90%)	1 block (e.g., 10%)

Common Pitfalls

Mistake: Reusing the same data for both training and testing the algorithm. Avoid: Always test the algorithm on data it was not trained on.
Mistake: Assuming a single 75/25 split is the optimal way to divide the dataset. Avoid: Use cross-validation to ensure all data points contribute to testing.
Mistake: Confusing parameter estimation with tuning parameter selection. Avoid: Use CV to find the best value for guessed tuning parameters (e.g., Ridge Regression).
Mistake: Using a small number of folds (e.g., 2-fold) for cross-validation. Avoid: Use 10-fold CV for robust results, or LOOCV if the dataset is tiny.

FAQs

Why is reusing training data for testing a bad idea? The algorithm will appear overly accurate because it has memorized the training data. This method fails to generalize to new, unseen data.
What is the difference between 4-fold and 10-fold cross-validation? 4-fold divides the data into four blocks, testing on 25% at a time. 10-fold divides it into ten blocks, testing on 10% at a time.
How do I decide which machine learning method to use? Use cross-validation to compare the performance (e.g., accuracy) of different methods on the test blocks. Select the method with the best summarized performance.
Is Leave-One-Out CV always better since it tests every sample individually? LOOCV is computationally expensive and often yields results with high variance. 10-fold CV is generally preferred in practice.

🧪 Stratified, Group-Aware, and Time-Based Cross-Validation

Plain k-fold is only the starting point — match the splitter to the data:

StratifiedKFold (classification): preserves the class ratio in every fold, so rare classes appear in each validation slice. Default for churn, fraud, and diagnosis tasks.
GroupKFold / StratifiedGroupKFold: keeps all rows of one entity (user, patient, device) in a single fold. Without it, the model "recognizes" entities instead of generalizing.
TimeSeriesSplit: rolling-origin splits for temporal data — always train on the past, validate on the future. Never shuffle timestamps.
Leakage rule: any preprocessing (impute/scale/encode/select) goes inside the CV loop via Pipeline, or validation scores are optimistic.

Common pitfalls

Mistake: Reusing the same data for both training and testing the algorithm. Avoid: Always test the algorithm on data it was not trained on.
Mistake: Assuming a single 75/25 split is the optimal way to divide the dataset. Avoid: Use cross-validation to ensure all data points contribute to testing.
Mistake: Confusing parameter estimation with tuning parameter selection. Avoid: Use CV to find the best value for guessed tuning parameters (e.g., Ridge Regression).
Mistake: Using a small number of folds (e.g., 2-fold) for cross-validation. Avoid: Use 10-fold CV for robust results, or LOOCV if the dataset is tiny.

StatQuest! Check it out. Talking about machine learning, yeah. StatQuest! Check it out. Talking about cross-validation, yeah. StatQuest! Hello, I'm Josh Starmer, and welcome to StatQuest. Today we're going to talk about cross-validation and it's going to be clearly explained. Okay, let's start with some data. We want to use the variables, chest pain, good blood circulation, etc., to predict if someone has heart disease. Then, when a new patient shows up, we can measure these variables and predict if they have heart disease or not. However, first we have to decide which machine learning method would be best. We could use logistic regression or K-nearest neighbors or support vector machines, and many more machine learning methods. How do we decide which one to use? Cross-validation allows us to compare different machine learning methods and get a sense of how well they will work in practice. Imagine that this blue column represented all of the data that we have collected about people with and without heart disease. We need to do two things with this data. One, we need to estimate the parameters for the machine learning methods. In other words, to use logistic regression, we have to use some of the data to estimate the shape of this curve. In machine learning lingo, estimating parameters is called training the algorithm. The second thing we need to do with this data is evaluate how well the machine learning methods work. In other words, we need to find out if this curve will do a good job categorizing new data. In machine learning lingo, evaluating a method is called testing the algorithm. Thus, using machine learning lingo, we need the data to: one, train the machine learning methods, and two, test the machine learning methods. A terrible approach would be to use all the data to estimate the parameters, i.e., to train the algorithm. Because then we wouldn't have any data left to test the method. Reusing the same data for both training and testing is a bad idea because we need to know how the method will work on data it wasn't trained on. A slightly better idea would be to use the first 75% of the data for training and the last 25% of the data for testing. We could then compare methods by seeing how well each one categorized the test data. But how do we know that using the first 75% of the data for training and the last 25% of the data for testing is the best way to divide up the data? What if we use the first 25% of the data for testing? Or what about one of these middle blocks? Rather than worry too much about which block would be best for testing, cross-validation uses them all, one at a time, and summarizes the results at the end. For example, cross-validation would start by using the first three blocks to train the method and then use the last block to test the method. And then it keeps track of how well the method did with the test data. Then it uses this combination of blocks to train the method. And this block is used for testing. And then it keeps track of how well the method did with the test data. Etc., etc., etc. In the end, every block of data is used for testing, and we can compare methods by seeing how well they performed. In this case, since the support vector machine did the best job classifying the test datasets, we'll use it. Bam! Note: In this example, we divided the data into four blocks. This is called four-fold cross-validation. However, the number of blocks is arbitrary. In an extreme case, we could call each individual patient or sample a block. This is called leave-one-out cross-validation. Each sample is tested individually. That said, in practice, it is very common to divide the data into 10 blocks. This is called 10-fold cross-validation. Double bam! One last note before we're done. Say like we wanted to use a method that involved a tuning parameter, a parameter that isn't estimated but is just sort of guessed. For example, ridge regression has a tuning parameter. Then we could use 10-fold cross-validation to help find the best value for that tuning parameter. Tiny bam. Hooray! We've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, well, please click the like button down below and consider buying one of my original songs. All right, then till next time, quest on.

Frequently asked questions

Why is reusing training data for testing a bad idea?

The algorithm will appear overly accurate because it has memorized the training data. This method fails to generalize to new, unseen data.

What is the difference between 4-fold and 10-fold cross-validation?

4-fold divides the data into four blocks, testing on 25% at a time. 10-fold divides it into ten blocks, testing on 10% at a time.

How do I decide which machine learning method to use?

Use cross-validation to compare the performance (e.g., accuracy) of different methods on the test blocks. Select the method with the best summarized performance.

Is Leave-One-Out CV always better since it tests every sample individually?

LOOCV is computationally expensive and often yields results with high variance. 10-fold CV is generally preferred in practice.