Gradient Boosting — XGBoost, LightGBM, CatBoost

Q: Why is the difference called a "pseudo residual"?

It reminds us that we are using Gradient Boost, not standard linear regression, where the term residual originates.

Q: Why does Gradient Boost build trees to predict residuals instead of the original weight?

By predicting the error (residual), the new tree takes a step toward correcting the previous prediction's mistake.

Q: What is the benefit of using a small learning rate?

Taking lots of small steps in the right direction results in better predictions and lower variance on testing data.

Q: How does Gradient Boost stop building trees?

It stops when it reaches the maximum number of trees specified or when additional trees fail to significantly reduce the residuals.

The current dominant family for tabular ML. The intuition behind GBM and why the three libraries exist. FIND_VIDEO: search 'gradient boosting xgboost lightgbm explained' — recommended channel: StatQuest / 3Blue1Brown. Aim for 12 min or under.

16 minutesVideo LessonPDF notes

Watch Gradient Boosting — XGBoost, LightGBM, CatBoost on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Lesson Context — The lesson introduces Gradient Boost for regression, assuming knowledge of Decision Trees and AdaBoost.
AdaBoost vs Gradient Boost Start — The comparison shows AdaBoost starts with a stump while Gradient Boost starts with a single leaf.
Initial Prediction — Gradient Boost’s initial prediction is the average value of the continuous variable being predicted.
Calculate Pseudo Residuals — Pseudo residuals are calculated as the difference between the observed values and the initial average prediction.
Tree Predicts Residuals — A decision tree is built using the features to predict the calculated pseudo residuals.
Apply Learning Rate — A learning rate scales the new tree's contribution to prevent overfitting and reduce variance.
New Residuals Calculated — New pseudo residuals are calculated based on the updated prediction, showing a small step in the right direction.
Iterative Improvement — Subsequent trees are built to predict the new residuals, scaled by the learning rate, and added to the model.
Final Prediction Summary — The final prediction is the sum of the initial leaf and all subsequent scaled trees.

PDF notes

Bottom line: You will be able to describe the step-by-step process of Gradient Boost for regression. You will also be able to compare and contrast its core mechanics, like tree size and scaling, with AdaBoost.

This lesson on Gradient Boosting — XGBoost, LightGBM, CatBoost is hands-on and example-driven. You will be able to describe the step-by-step process of Gradient Boost for regression. You will also be able to compare and contrast its core mechanics, like tree size and scaling, with AdaBoost.

What You'll Be Able To Do

Explain the role of the initial leaf in Gradient Boost regression.
Calculate the pseudo residual used to train subsequent trees.
Contrast how Gradient Boost and AdaBoost scale the contribution of new models.
Describe the purpose of the learning rate in controlling model variance.
Justify why Gradient Boost builds trees to predict residuals, not the target variable.
Identify the common configuration parameters for tree size in practice.

Topics Covered in Gradient Boosting — XGBoost, LightGBM, CatBoost

Lesson Context (0:00 - 0:30) — The lesson introduces Gradient Boost for regression, assuming knowledge of Decision Trees and AdaBoost.
AdaBoost vs Gradient Boost Start (1:50 - 2:40) — The comparison shows AdaBoost starts with a stump while Gradient Boost starts with a single leaf.
Initial Prediction (3:00 - 3:45) — Gradient Boost’s initial prediction is the average value of the continuous variable being predicted.
Calculate Pseudo Residuals (4:00 - 5:15) — Pseudo residuals are calculated as the difference between the observed values and the initial average prediction.
Tree Predicts Residuals (5:20 - 6:30) — A decision tree is built using the features to predict the calculated pseudo residuals.
Apply Learning Rate (6:40 - 7:45) — A learning rate scales the new tree's contribution to prevent overfitting and reduce variance.
New Residuals Calculated (8:00 - 9:15) — New pseudo residuals are calculated based on the updated prediction, showing a small step in the right direction.
Iterative Improvement (9:30 - 10:45) — Subsequent trees are built to predict the new residuals, scaled by the learning rate, and added to the model.
Final Prediction Summary (11:00 - 11:45) — The final prediction is the sum of the initial leaf and all subsequent scaled trees.

SQL Cheat Sheet

Gradient Boost Regression — Predicts continuous values by iteratively correcting previous errors
```
SELECT AVG(weight) FROM training_data;
```
Initial Leaf — The starting prediction, typically the average of the target variable
```
SELECT AVG(weight) FROM training_data;
```
Pseudo Residual — Difference between observed value and the current predicted value
```
SELECT observed_weight - predicted_weight AS pseudo_residual FROM data;
```
Learning Rate — Scales the contribution of each new tree to reduce variance
Tree Size Restriction — Limits the maximum number of leaves (e.g., 8 to 32) per tree
AdaBoost — Builds short trees (stumps) and weights them based on performance
```
SELECT * FROM training_data WHERE error_rate < 0.5;
```

Comparison Table

Feature	AdaBoost	Gradient Boost
Initial Model	Short tree (stump)	Single leaf (average value)
Tree Size	Usually a stump	Fixed size (often 8 to 32 leaves)
Tree Scaling	Scaled based on error compensation	All trees scaled by fixed learning rate
Strategy	Focus on hard-to-classify samples	Take many small steps

Common Pitfalls

Mistake: Confusing Gradient Boost regression with standard linear regression. Avoid: Remember Gradient Boost uses iterative trees to predict residuals.
Mistake: Allowing trees to fit the residuals perfectly without scaling. Avoid: Always apply a learning rate (e.g., 0.1) to control variance.
Mistake: Assuming all boosting algorithms use the same scaling method. Avoid: AdaBoost scales based on performance; Gradient Boost scales all equally.
Mistake: Thinking the first tree predicts the target variable directly. Avoid: The first tree predicts the pseudo residuals from the initial average guess.

FAQs

Why is the difference called a "pseudo residual"? It reminds us that we are using Gradient Boost, not standard linear regression, where the term residual originates.
Why does Gradient Boost build trees to predict residuals instead of the original weight? By predicting the error (residual), the new tree takes a step toward correcting the previous prediction's mistake.
What is the benefit of using a small learning rate? Taking lots of small steps in the right direction results in better predictions and lower variance on testing data.
How does Gradient Boost stop building trees? It stops when it reaches the maximum number of trees specified or when additional trees fail to significantly reduce the residuals.

🔍 Model Explainability: SHAP Values (+ LIME)

Boosted trees win on accuracy but must still explain themselves — to risk teams, regulators, and debugging engineers:

SHAP (SHapley Additive exPlanations) attributes each prediction to input features with game-theoretic consistency: global importance (summary_plot), local reasons (waterfall per row), and interaction effects. shap.TreeExplainer is exact and fast for XGBoost/LightGBM.
LIME fits a local surrogate around one prediction — useful for text/tabular spot-checks, but less consistent than SHAP across rows.
Bias-audit workflow: slice SHAP importance by sensitive segment; a feature that drives approvals for one group and denials for another is a finding, not a feature. Document it in the model card.

Common pitfalls

Mistake: Confusing Gradient Boost regression with standard linear regression. Avoid: Remember Gradient Boost uses iterative trees to predict residuals.
Mistake: Allowing trees to fit the residuals perfectly without scaling. Avoid: Always apply a learning rate (e.g., 0.1) to control variance.
Mistake: Assuming all boosting algorithms use the same scaling method. Avoid: AdaBoost scales based on performance; Gradient Boost scales all equally.
Mistake: Thinking the first tree predicts the target variable directly. Avoid: The first tree predicts the pseudo residuals from the initial average guess.

Gradient Boost Part 1 Regression Main Ideas StatQuest Hello! I'm Josh Starmer and welcome to StatQuest. Today, we're going to talk about the Gradient Boost machine learning algorithm. Specifically, we're going to focus on how Gradient Boost is used for regression. Note, this StatQuest assumes you already understand Decision Trees. So if you're not already down with those, check out the quest. This StatQuest also assumes that you are familiar with AdaBoost and the tradeoff between bias and variance. If not, check out the quests. The links are in the description below. This StatQuest is the first part in a series that explains how the Gradient Boost machine learning algorithm works. Specifically, we'll use this data, where we have the height measurements from six people, their favorite colors, their genders, and their weights. And we'll walk through step-by-step the most common way that Gradient Boost fits a model to this training data. Note, when Gradient Boost is used to predict a continuous value, like weight, we say that we are using Gradient Boost for regression. Using Gradient Boost for regression is different from doing linear regression. So while the two methods are related, don't get them confused with each other. Part 2 in this series will dive deep into the math behind the Gradient Boost algorithm for regression, walking through it step-by-step and proving that what we covered today is correct. Part 3 in this series shows how Gradient Boost can be used for classification. Specifically, we'll walk through step-by-step the most common way Gradient Boost can classify someone as either loving the movie Troll 2 or not loving Troll 2. Part 4 will return to the math behind Gradient Boost, this time focusing on classification, walking through it step-by-step. Note, the Gradient Boost algorithm looks complicated because it was designed to be configured in a wide variety of ways. But the reality is that 99% of the time, only one configuration is used to predict continuous values like weight, and one configuration is used to classify samples into different categories. This StatQuest focuses on showing you the most common way Gradient Boost is used to predict a continuous value like weight. If you are familiar with AdaBoost, then a lot of Gradient Boost will seem very similar. So let's briefly compare and contrast AdaBoost and Gradient Boost. If we want to use these measurements to predict weight, then AdaBoost starts by building a very short tree, called a stump, from the training data. And then the amount of say that the new stump has on the final output is based on how well it compensated for those previous errors. Then AdaBoost builds the next stump based on errors that the previous stump made. In this example, the new stump did a poor job compensating for the previous stump's errors, and its size reflects its reduced amount of say. Then AdaBoost builds another stump based on the errors made by the previous stump. And this stump did a little better than the last stump, so it's a little larger. Then AdaBoost continues to make stumps in this fashion until it has made the number of stumps you asked for or it has a perfect fit. In contrast, Gradient Boost starts by making a single leaf instead of a tree or stump. This leaf represents an initial guess for the weights for all of the samples. When trying to predict a continuous value like weight, the first guess is the average value. Then Gradient Boost builds a tree. Like AdaBoost, this tree is based on the errors made by the previous tree. But unlike AdaBoost, this tree is usually larger than a stump. That said, Gradient Boost still restricts the size of the tree. In the simple example that we will go through in this StatQuest, we will build trees with up to four leaves, but no larger. However, in practice, people often set the maximum number of leaves to be between 8 and 32. Thus, like AdaBoost, Gradient Boost builds fixed size trees based on the previous tree's errors. But unlike AdaBoost, each tree can be larger than a stump. Also, like AdaBoost, Gradient Boost scales the trees. However, Gradient Boost scales all trees by the same amount. Then Gradient Boost builds another tree based on the errors made by the previous tree. And then it scales the tree. And Gradient Boost continues to build trees in this fashion until it has made the number of trees you asked for or additional trees fail to improve the fit. Now that we know the main similarities and differences between Gradient Boost and AdaBoost, let's see how the most common Gradient Boost configuration would use this training data to predict weight. The first thing we do is calculate the average weight. This is the first attempt at predicting everyone's weight. In other words, if we stopped right now, we would predict that everyone weighed 71.2 kg. However, Gradient Boost doesn't stop here. The next thing we do is build a tree based on the errors from the first tree. The errors that the previous tree made are the differences between the observed weights and the predicted weight, 71.2. So let's start by plugging in 71.2 for the predicted weight. And then plug in the first observed weight. And do the math. And save the difference, which is called a pseudo residual, in a new column. Note, the term pseudo residual is based on linear regression, where the difference between the observed values and the predicted values results in residuals. The pseudo part of pseudo residual is a reminder that we are doing Gradient Boost, not linear regression, and is something I'll talk more about in part 2 of this series when we go through the math. Now we do the same thing for the remaining weights. Now we will build a tree using height, favorite color, and gender to predict the residuals. If it seems strange to predict the residuals instead of the original weight, just bear with me and soon all will become clear. So, setting aside the reason why we are building a tree to predict the residuals for the time being, here's the tree. Remember, in this example, we are only allowing up to four leaves. But when using a larger data set, it is common to allow anywhere from 8 to 32. By restricting the total number of leaves, we get fewer leaves than residuals. As a result, these two rows of data go to the same leaf. So we replace these residuals with their average. And these two rows of data go to the same leaf. So we replace these residuals with their average. Now we can combine the original leaf with the new tree to make a new prediction of an individual's weight from the training data. We start with the initial prediction, 71.2. Then we run the data down the tree, and we get 16.8. So the predicted weight equals 71.2 + 16.8, which equals 88. Which is the same as the observed weight. Is this awesome? No. The model fits the training data too well. In other words, we have low bias, but probably very high variance. Gradient Boost deals with this problem by using a learning rate to scale the contribution from the new tree. The learning rate is a value between 0 and 1. In this case, we'll set the learning rate to 0.1. Now the predicted weight equals 71.2 + 0.1 * 16.8, which equals 72.9. With the learning rate set to 0.1, the new prediction isn't as good as it was before. But it's a little better than the prediction made with just the original leaf, which predicted that all samples would weigh 71.2. In other words, scaling the tree by the learning rate results in a small step in the right direction. According to the dude that invented Gradient Boost, Jerome Friedman, empirical evidence shows that taking lots of small steps in the right direction results in better predictions with a testing data set, i.e., lower variance. Bam! So let's build another tree so we can take another small step in the right direction. Just like before, we calculate the pseudo residuals, the difference between the observed weights and our latest predictions. So we plug in the observed weight, and the new predicted weight. And we get 15.1. And we save that in the column for pseudo residuals. Then we repeat for all of the other individuals in the training data set. Small bam. Note, these are the original residuals from when our prediction was simply the average overall weight. And these are the residuals after adding the new tree scaled by the learning rate. The new residuals are all smaller than before, so we've taken a small step in the right direction. Double bam. Now let's build a new tree to predict the new residuals. And here's the new tree. Note, in this simple example, the branches are the same as before. However, in practice, the trees can be different each time. Just like before, since multiple samples ended up in these leaves, we just replace the residuals with their averages. Now we combine the new tree with the previous tree and the initial leaf. Note, we scale all of the trees by the learning rate, which we set to 0.1. And add everything together. Now we're ready to make a new prediction from the training data. Just like before, we start with the initial prediction. Then add the scaled amount from the first tree and the scaled amount from the second tree. That gives us 71.2 + 0.1 * 16.8 + 0.1 * 15.1, which equals 74.4. Which is another small step closer to the observed weight. Now we use the initial leaf plus the scaled values from the first tree plus the scaled values from the second tree to calculate new residuals. Remember, these were the residuals from when we just used a single leaf to predict weight. And these were the residuals after we added the first tree to the prediction. And these are the residuals after we added the second tree to the prediction. Each time we add a tree to the prediction, the residuals get smaller. So we've taken another small step towards making good predictions. Now we build another tree to predict the new residuals. And add it to the chain of trees that we have already created. And we keep making trees until we reach the maximum specified or adding additional trees does not significantly reduce the size of the residuals. Bam. Then, when we get some new measurements, we can predict weight by starting with the initial prediction, then adding the scaled value from the first tree and the second tree and the third tree, etc., etc., etc. And once the math is all done, we are left with the predicted weight. In this case, we predicted that this person weighed 70 kg. Triple bam. In summary, when Gradient Boost is used for regression, we start with a leaf that is the average value of the variable we want to predict. In this case, we wanted to predict weight. Then we add a tree based on the residuals, the difference between the observed values and the predicted values. And we scale the tree's contribution to the final prediction with a learning rate. Then we add another tree based on the new residuals. And we keep adding trees based on the errors made by the previous tree. That's all there is to it. Bam. Tune in for part 2 in this series, when we dive deep into the math behind the Gradient Boost algorithm for regression, walking through it step-by-step and proving that it really is this simple. Hooray! We've made it to the end of another exciting StatQuest! If you like this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, consider buying one of my original songs or buying a StatQuest T-shirt or hoodie. The links are in the description below. Alright, until next time, quest on!

Frequently asked questions

Why is the difference called a "pseudo residual"?