Back to Evaluation, Leakage, and Unsupervised

Data Leakage — The #1 ML Bug

Q: Why does data leakage cause models to fail in production?

Leakage causes the model to partially memorize test data features, leading to unrealistically high performance scores during evaluation. In production, this memorized information is absent, causing performance to drop significantly.

Q: How should I handle feature processing during cross-validation?

For every fold, the feature processing (like standardization) must be calculated only on the training data for that fold, and then applied to the validation data.

Q: What is the primary rule for avoiding data leakage?

Always split the data into training and testing sets first, and only then perform any feature preprocessing or transformation steps.

Q: What is 'time leakage'?

Time leakage occurs in time series data when future information is accidentally included in the training set, making prediction unrealistically easy.

The reason your model is too good to be true is almost always leakage. Spot it before deployment. FIND_VIDEO: search 'data leakage machine learning examples' — recommended channel: Kaggle / StatQuest. Aim for 11 min or under.

9 minutesVideo LessonPDF notes

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

ML Objective — The goal of machine learning is training a model to predict accurately on unseen data.
Defining Leakage — Data leakage occurs if the model trains on information that will not be available during production prediction.
TF-IDF Example — Computing TF-IDF on the entire corpus before splitting the data causes leakage.
Standardization Example — Calculating mean and standard deviation on the full dataset before splitting is a common leakage error.
The Solution — To avoid leakage, always split the data first, then apply transforms only on the training set.
Cross-Validation — Feature processing must be done within each fold of cross-validation after the validation data is split.
Duplicate Rows — Duplicate rows between train and test sets, often from oversampling, can cause training example leakage.
Group Leakage — Group leakage happens when related data points, like multiple X-rays from one patient, are split across train and test sets.
Time Leakage — In time series data, mixing future data into the training set creates time leakage.
Summary & Impact — Leakage causes the model to overestimate its utility, leading to failure when deployed in a production environment.

PDF notes

Bottom line: You will be able to identify and prevent various types of data leakage that cause machine learning models to overestimate their performance. You will learn the critical best practice: always split your data into train/test sets before applying any feature preprocessing or transformation.

This lesson on Data Leakage — The #1 ML Bug is hands-on and example-driven. You will be able to identify and prevent various types of data leakage that cause machine learning models to overestimate their performance. You will learn the critical best practice: always split your data into train/test sets before applying any feature preprocessing or transformation.

What You'll Be Able To Do

Define data leakage and its impact on model utility.
Identify premature featurization as a primary source of leakage.
Apply the correct sequence of steps for data splitting and transformation.
Recognize specific types of leakage, including group and time leakage.
Explain why cross-validation requires feature processing within each fold.

Topics Covered in Data Leakage — The #1 ML Bug

ML Objective (0:00 - 0:30) — The goal of machine learning is training a model to predict accurately on unseen data.
Defining Leakage (0:30 - 1:00) — Data leakage occurs if the model trains on information that will not be available during production prediction.
TF-IDF Example (1:00 - 2:30) — Computing TF-IDF on the entire corpus before splitting the data causes leakage.
Standardization Example (2:30 - 3:50) — Calculating mean and standard deviation on the full dataset before splitting is a common leakage error.
The Solution (3:50 - 4:30) — To avoid leakage, always split the data first, then apply transforms only on the training set.
Cross-Validation (4:30 - 5:30) — Feature processing must be done within each fold of cross-validation after the validation data is split.
Duplicate Rows (5:30 - 6:00) — Duplicate rows between train and test sets, often from oversampling, can cause training example leakage.
Group Leakage (6:00 - 7:00) — Group leakage happens when related data points, like multiple X-rays from one patient, are split across train and test sets.
Time Leakage (7:00 - 8:00) — In time series data, mixing future data into the training set creates time leakage.
Summary & Impact (8:00 - 8:40) — Leakage causes the model to overestimate its utility, leading to failure when deployed in a production environment.

SQL Cheat Sheet

Data Leakage — Training model uses information unavailable during production prediction
Premature Featurization — Applying transforms before splitting data into train and test sets
```
SELECT TFIDF(corpus) FROM documents;
```
Standardization — Normalizing features using mean and standard deviation
```
SELECT (X - AVG(X)) / STDDEV(X) FROM features;
```
Train-Test Split — Separating data into sets for model development and evaluation
```
SELECT * FROM data WHERE split = 'train';
```
Group Leakage — Data from the same group appears in both training and test splits
```
SELECT * FROM xrays WHERE patient_id = 123;
```
Time Leakage — Future data is included in the training set for time series
```
SELECT * FROM stock_data WHERE date < '2023-01-01';
```

Comparison Table

Process Step	Correct (Avoids Leakage)	Incorrect (Causes Leakage)
Order of Operations	Split first, then transform.	Transform first, then split.
Statistic Calculation	Calculate mean/std dev only on Train.	Calculate mean/std dev on entire data.
Resulting Model Utility	Accurate estimate of performance.	Overestimated utility; fails in production.

Common Pitfalls

Mistake: Computing standardization statistics (mean/std dev) on the entire dataset. Avoid: Calculate statistics only on the training set, then apply them to the test set.
Mistake: Calculating TF-IDF across the entire document corpus before splitting. Avoid: Compute TF-IDF only on the training documents after the split is complete.
Mistake: Allowing duplicate rows between the training and validation sets. Avoid: Ensure data augmentation or upsampling is done carefully to prevent cross-split duplicates.
Mistake: Using random splitting on non-IID data like patient X-rays. Avoid: Ensure all data points belonging to a single group stay within one split (train or test).

FAQs

Why does data leakage cause models to fail in production? Leakage causes the model to partially memorize test data features, leading to unrealistically high performance scores during evaluation. In production, this memorized information is absent, causing performance to drop significantly.
How should I handle feature processing during cross-validation? For every fold, the feature processing (like standardization) must be calculated only on the training data for that fold, and then applied to the validation data.
What is the primary rule for avoiding data leakage? Always split the data into training and testing sets first, and only then perform any feature preprocessing or transformation steps.
What is 'time leakage'? Time leakage occurs in time series data when future information is accidentally included in the training set, making prediction unrealistically easy.

🛡️ Leakage-Prevention Checklist (Temporal + Group-Aware Splits)

Leakage is the #1 reason offline metrics lie. Harden every evaluation:

Split first, preprocess later. Fit imputers, scalers, encoders, and target statistics on the TRAIN fold only — never on the full frame. In sklearn, enforce this structurally with Pipeline + ColumnTransformer.
Respect time. For temporal data use time-based splits (train on past, validate on future). Random k-fold on timestamped rows leaks the future.
Respect groups. When rows share an entity (user, account, device), use GroupKFold / StratifiedGroupKFold so no entity spans train and valid.
Audit features. Any feature computed with knowledge of the label or the future (e.g. "days until churn", aggregates over the full window) is suspect — drop it or recompute point-in-time.

Common pitfalls

Mistake: Computing standardization statistics (mean/std dev) on the entire dataset. Avoid: Calculate statistics only on the training set, then apply them to the test set.
Mistake: Calculating TF-IDF across the entire document corpus before splitting. Avoid: Compute TF-IDF only on the training documents after the split is complete.
Mistake: Allowing duplicate rows between the training and validation sets. Avoid: Ensure data augmentation or upsampling is done carefully to prevent cross-split duplicates.
Mistake: Using random splitting on non-IID data like patient X-rays. Avoid: Ensure all data points belonging to a single group stay within one split (train or test).

This is Ritesh Srinivasan and welcome to my channel. In this video, let's look at what is the concept of data leakage or target leakage in machine learning. So what is the objective of machine learning? It is to develop a machine learning model. Right? One of the objectives. Okay, so how do you do it? You make use of some data, right? And you do training, right? And then what do you do? You make use of this model to predict on unseen data. Right? So that is what is the objective of developing a machine learning model. Now, in this process, if your machine learning model, when it is training, right? When it is training, if it gets access to information which may not be present during prediction on unseen data, you get data leakage. Okay? So to make it more clear, I'll just give you an example. Okay? So you have text data. Right? So you have text data. Okay? So basically you have a set of documents. Okay? So which is your corpus. Okay? So then what do you do? You want to develop a classification algorithm. Okay? Text classification model, basically. Okay? That you want because you want to predict on some unseen documents. Okay? Basically you want to classify unseen documents. So for classification we need to compute a set of features. Right? To train a ML model. So for text typically we might compute TF-IDF. Right? Now, if TF-IDF were being computed on the entire corpus, right? Then you would have data leakage. Okay? If you compute on the entire corpus TF-IDF and then do a split say train, test, okay? Then you would have data leakage. This is because, uh, if you have done TF-IDF on the entire corpus, okay? And then, uh, you have split that features into train and test. So some unseen features, okay? Test data should have been unseen data, right? But some of those, say terms which are present in the test data may also be, you know, it will come over here in the train data, because of which a model developed on this train data may perform very well on this test data. But then when you put into production, what will happen is that it may fail. Okay? So this is one example. Right? So the next example is suppose you have numeric features, right? You have features X1, X2, X3 and so on on your data, right? And what you are doing is that you are doing standardization, right? On this data. Okay? So when you are doing standardization you will compute mean and standard deviation. Okay? On this data. Right? Now, if you do this mean and standard deviation on the entire data and then use split your data into train and test, okay? Here also you have data leakage. Okay? So what should you do over here? You should actually compute mean and standard deviation only on train and use that same mean and standard deviation on the test to normalize your data. Okay? So to avoid data leakage what we should do is that we should always do our transforms on the data after the split, right? So when you are doing preprocessing, split the data first, right? And then do your preprocessing. Okay? So you do your, uh, say standardization or TF-IDF computation or any other kind of feature processing on the train data only. Right? And fit that to test data. So that is how you can avoid data leakage. So what I explained about data leakage is was this training example leakage. Okay? I talked about premature featurization. So basically, uh, you know, leaking from premature featurization before the split, right? So even in cross-validation, ideally what you should do is that for every fold, when you are doing n-fold cross-validation, for every fold you should, uh, you know, once the validation data is split and then you have the training data, on the training data you have to do your, uh, feature standardization or say TF-IDF computation, any kind of feature preprocessing, right? So that needs to be done in cross-validation basically. Okay? In case of your normal train-test split also, first you should split the data into train and test dataset and then on the train only you have to apply your transformer transforms and then fit them on the test data. Okay? There is also another problem of duplicate rows, uh, between the train and validation test. This could be because of oversampling, uh, dataset to pad its size before splitting or different rotations, augmentations of the same image, right? Um, or it could be duplicating rows to upsample the minority class. So this also can cause training example leakage or data leakage. Okay? In case of non-IID data like, uh, time series data, you can have time leakage. You can also have another thing called group leakage because when you are splitting you are not properly splitting on a grouping column, right? So there is this example where you had 100k x-rays of 30k patients, meaning three images per patient. So when the, uh, the particular paper used random splitting instead of ensuring that all, uh, images of a patient was in the same split. When random splitting was done on this, what can happen is that some patients' images can be in both train and test. And because of which the model kind of partially memorized the patients instead of learning to recognize pneumonia in chest x-rays. So this is some kind of a group leakage. Okay? And if you look at time dependent dataset, um, for, uh, which the structure of the system being studied evolves over time, right? Like your stock market predicting stock values, so if the data is trained on data for a certain five-year-old period, it is unrealistic to treat the subsequent five-year period as a draw from the same population. Okay? Another example over here is that when you are creating your features or other things, if you have one year's data and you assume that you want to use the first six months data for training and the next six months data for predicting. In case you mix up this, right? You might have first month's data as well as seven months data in your training data, which can cause problems. Okay? So this was about data leakage, right? Or leakage. It is essentially the use of information in the model training process which would not be expected to be available at prediction time. It causes the predictive scores or the model to overestimate the model's utility. Okay? That is what happens when it runs in a production environment. Okay? So this is about the concept of data leakage and it is a very important concept. Okay? So the best practices is always to avoid data leakage is to actually split your data before you perform any of your feature processing. I hope this video on leakage, the concept of data leakage in machine learning is useful for you. If you like the video, please like, share, subscribe to the channel. See you in another video. Happy learning.

Frequently asked questions

Why does data leakage cause models to fail in production?