Decision Trees and Random Forests

The model that dominates tabular ML. Trees, then forests, then why bagging works. FIND_VIDEO: search 'decision tree random forest tutorial' — recommended channel: StatQuest. Aim for 11 min or under.

10 minutesVideo LessonPDF notes

Watch Decision Trees and Random Forests on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Decision Tree Limitations — Decision trees are easy to use but are inaccurate for predictive learning on new samples.
Random Forest Advantage — Random Forests combine the simplicity of trees with flexibility, resulting in vastly improved accuracy.
Step 1: Bootstrapping Data — A bootstrapped dataset is created by randomly selecting samples from the original data with replacement.
Step 2: Building Trees — Build a decision tree using the bootstrapped data, but only consider a random subset of variables at each split.
Forest Variety — Repeating the process creates a wide variety of trees, which makes the Random Forest effective.
Using the Forest (Voting) — To classify a new sample, run it through all trees and use the majority vote for the final prediction.
Terminology: Bagging — The process of bootstrapping data and aggregating results via voting is called bagging.
Out-of-Bag Dataset — The Out-of-Bag (OOB) dataset consists of samples not used to build a specific tree, typically about one-third of the data.
Estimating OOB Error — OOB samples are used to measure accuracy, and the proportion incorrectly classified is the OOB error.
Optimizing Variables — Optimize the forest by comparing OOB error rates across different numbers of variables considered per step.

PDF notes

Bottom line: You will be able to construct and evaluate a Random Forest model by understanding the roles of bootstrapping and variable subsetting. You will also learn how to use Out-of-Bag error estimation to determine the model's accuracy and optimize its hyperparameters.

This lesson on Decision Trees and Random Forests is hands-on and example-driven. You will be able to construct and evaluate a Random Forest model by understanding the roles of bootstrapping and variable subsetting. You will also learn how to use Out-of-Bag error estimation to determine the model's accuracy and optimize its hyperparameters.

What You'll Be Able To Do

Explain why Random Forests are generally more accurate than single Decision Trees.
Describe the process of creating a bootstrapped dataset from original data.
Apply the concept of variable subsetting when building individual trees in a forest.
Define "bagging" in the context of ensemble modeling.
Calculate the prediction for a new sample using the voting mechanism of a Random Forest.
Evaluate the accuracy of a Random Forest using the Out-of-Bag error rate.

Topics Covered in Decision Trees and Random Forests

Decision Tree Limitations (0:00 - 0:35) — Decision trees are easy to use but are inaccurate for predictive learning on new samples.
Random Forest Advantage (0:35 - 1:05) — Random Forests combine the simplicity of trees with flexibility, resulting in vastly improved accuracy.
Step 1: Bootstrapping Data (1:05 - 2:20) — A bootstrapped dataset is created by randomly selecting samples from the original data with replacement.
Step 2: Building Trees (2:20 - 4:20) — Build a decision tree using the bootstrapped data, but only consider a random subset of variables at each split.
Forest Variety (4:20 - 5:00) — Repeating the process creates a wide variety of trees, which makes the Random Forest effective.
Using the Forest (Voting) (5:00 - 6:20) — To classify a new sample, run it through all trees and use the majority vote for the final prediction.
Terminology: Bagging (6:20 - 6:40) — The process of bootstrapping data and aggregating results via voting is called bagging.
Out-of-Bag Dataset (6:40 - 7:45) — The Out-of-Bag (OOB) dataset consists of samples not used to build a specific tree, typically about one-third of the data.
Estimating OOB Error (7:45 - 9:20) — OOB samples are used to measure accuracy, and the proportion incorrectly classified is the OOB error.
Optimizing Variables (9:20 - 10:20) — Optimize the forest by comparing OOB error rates across different numbers of variables considered per step.

SQL Cheat Sheet

Decision Tree — Simple model that struggles with classifying new samples accurately
```
SELECT * FROM training_data WHERE feature_A > 5;
```
Random Forest — Ensemble method combining many trees for vastly improved accuracy
```
SELECT prediction FROM forest_model WHERE patient_id = 101;
```
Bootstrapping — Creating same-size dataset by sampling original data with replacement
```
SELECT sample_data FROM original_data ORDER BY RANDOM() LIMIT 4;
```
Variable Subsetting — Randomly selecting a few features to consider at each node split
```
SELECT feature_1, feature_2 FROM available_features LIMIT 2;
```
Bagging — Bootstrapping data and aggregating tree predictions via voting
Out-of-Bag (OOB) Dataset — Data samples not included in a specific bootstrapped dataset (approx 1/3)
```
SELECT COUNT(*) FROM oob_samples WHERE prediction <> actual;
```

Comparison Table

Characteristic	Decision Tree	Random Forest
Core Structure	Single tree model	Ensemble of many trees
Data Used	Entire training dataset	Bootstrapped subsets of data
Feature Selection	Considers all variables	Random subset of variables
Accuracy on New Data	Inaccurate (not flexible)	Vastly improved (flexible)

Common Pitfalls

Mistake: Assuming a single Decision Tree is sufficient for predictive learning. Avoid: Use Random Forests or other ensemble methods for better generalization and accuracy.
Mistake: Forgetting to sample with replacement when creating bootstrapped datasets. Avoid: Ensure samples can be picked more than once to maintain the original dataset size.
Mistake: Using the same set of variables for splitting every node in a tree. Avoid: Randomly select a subset of variables at every single node split.
Mistake: Using data that built a tree to estimate its accuracy (overfitting). Avoid: Use the Out-of-Bag (OOB) samples to estimate accuracy reliably.

FAQs

Why are Decision Trees "not that awesome" in practice? They work great on the data used to create them, but they lack flexibility and are inaccurate when classifying new, unseen samples.
What is the key difference between bootstrapping and standard sampling? Bootstrapping involves sampling with replacement, meaning the same sample can be selected multiple times to create the new dataset.
What is "bagging"? Bagging is the combination of bootstrapping the data and then aggregating the results (like voting) from the resulting models to make a final decision.
How many variables should I consider at each split? A common starting point is the square root of the total number of variables; then test settings above and below this value.
What proportion of data usually ends up in the Out-of-Bag dataset? Typically, about one-third (1/3) of the original data is not included in any specific bootstrapped dataset.

Common pitfalls

Mistake: Assuming a single Decision Tree is sufficient for predictive learning. Avoid: Use Random Forests or other ensemble methods for better generalization and accuracy.
Mistake: Forgetting to sample with replacement when creating bootstrapped datasets. Avoid: Ensure samples can be picked more than once to maintain the original dataset size.
Mistake: Using the same set of variables for splitting every node in a tree. Avoid: Randomly select a subset of variables at every single node split.
Mistake: Using data that built a tree to estimate its accuracy (overfitting). Avoid: Use the Out-of-Bag (OOB) samples to estimate accuracy reliably.

Wandering around a random forest, I won't get lost because of StatQuest. Hello, I'm Josh Starmer and welcome to StatQuest. Today we're going to be starting part one of a series on random forests and we're going to talk about building and evaluating random forests. Note, random forests are built from decision trees, so if you don't already know about those, check out my StatQuest. Bam! Decision trees are easy to build, easy to use, and easy to interpret. But in practice, they are not that awesome. To quote from The Elements of Statistical Learning, aka the Bible of machine learning, "Trees have one aspect that prevents them from being the ideal tool for predictive learning, namely inaccuracy." In other words, they work great with the data used to create them, but they are not flexible when it comes to classifying new samples. The good news is that random forests combine the simplicity of decision trees with flexibility, resulting in a vast improvement in accuracy. So let's make a random forest. Step one: create a bootstrapped dataset. Imagine that these four samples are the entire dataset that we are going to build a tree from. I know it's crazy small, but just pretend for now. To create a bootstrapped dataset that is the same size as the original, we just randomly select samples from the original dataset. The important detail is that we're allowed to pick the same sample more than once. This is the first sample that we randomly select, so it's the first sample in our bootstrapped dataset. This is the second randomly selected sample from the original dataset, so it's the second sample in our bootstrapped dataset. Here's the third randomly selected sample. So here it is in the bootstrapped dataset. Lastly, here's the fourth randomly selected sample. Note: it's the same as the third. And here it is. Bam! We've created a bootstrapped dataset. Step two for creating a random forest is to create a decision tree using the bootstrapped dataset, but only use a random subset of variables or columns at each step. In this example, we will only consider two variables or columns at each step. Note: we'll talk more about how to determine the optimal number of variables to consider later. Thus, instead of considering all four variables to figure out how to split the root node, we randomly select two. In this case, we randomly selected "good blood circulation" and "blocked arteries" as candidates for the root node. Just for the sake of the example, assume that "good blood circulation" did the best job separating the samples. Since we used "good blood circulation," I'm going to gray it out so that we focus on the remaining variables. Now we need to figure out how to split samples at this node. Just like for the root, we randomly select two variables as candidates, instead of all three remaining columns. And we just build the tree as usual, but only considering a random subset of variables at each step. Double bam! We built a tree, one, using a bootstrapped dataset and, two, only considering a random subset of variables at each step. Here's the tree we just made. Now, go back to step one and repeat. Make a new bootstrapped dataset and build a tree considering a subset of variables at each step. Ideally, you do this hundreds of times, but we only have space to show six, but you get the idea. Using a bootstrapped sample and considering only a subset of variables at each step results in a wide variety of trees. The variety is what makes random forests more effective than individual decision trees. Sweet. Now that we've created a random forest, how do we use it? Well, first we get a new patient. We've got all the measurements and now we want to know if they have heart disease or not. So we take the data and run it down the first tree that we made. Boop boop boop boop boop boop. The first tree says yes, the patient has heart disease. And we keep track of that here. Now we run the data down the second tree that we made. The second tree also says yes. And we keep track of that here. And then we repeat for all the trees we made. After running the data down all of the trees in the random forest, we see which option received more votes. In this case, yes received the most votes, so we will conclude that this patient has heart disease. Bam! Oh no, terminology alert! Bootstrapping the data plus using the aggregate to make a decision is called bagging. Okay. Now we've seen how to create and use a random forest. How do we know if it's any good? Remember when we created the bootstrapped dataset? We allowed duplicate entries in the bootstrapped dataset. As a result, this entry was not included in the bootstrapped dataset. Typically, about 1/3 of the original data does not end up in the bootstrapped dataset. Here's the entry that didn't end up in the bootstrapped dataset. If the original dataset were larger, we'd have more than just one entry over here. This is called the out-of-bag dataset. If it were up to me, I would have named it the "out-of-boot" dataset, since it's the entries that didn't make it into the bootstrapped dataset. Unfortunately, it's not up to me. Since the out-of-bag data was not used to create this tree, we can run it through and see if it correctly classifies the sample as "no heart disease." In this case, the tree correctly labels the out-of-bag sample "no." Then we run this out-of-bag sample through all of the other trees that were built without it. This tree incorrectly labeled the out-of-bag sample "yes." These trees correctly labeled the out-of-bag sample "no." Since the label with the most votes wins, it is the label that we assign this out-of-bag sample. In this case, the out-of-bag sample is correctly labeled by the random forest. We then do the same thing for all of the other out-of-bag samples for all of the trees. This out-of-bag sample was also correctly labeled. This out-of-bag sample was incorrectly labeled. Et cetera, et cetera, et cetera. Ultimately, we can measure how accurate our random forest is by the proportion of out-of-bag samples that were correctly classified by the random forest. The proportion of out-of-bag samples that were incorrectly classified is the out-of-bag error. Okay, we now know how to, one, build a random forest, two, use a random forest, and, three, estimate the accuracy of a random forest. However, now that we know how to do this, we can talk a little more about how to do this. Remember when we built our first tree and we only used two variables (columns of data) to make a decision at each step? Now we can compare the out-of-bag error for a random forest built using only two variables per step to a random forest built using three variables per step. And we test a bunch of different settings and choose the most accurate random forest. In other words, one, we build a random forest, and then, two, we estimate the accuracy of a random forest. Then we change the number of variables used per step. And we do this a bunch of times and then choose the one that is the most accurate. Typically, we start by using the square of the number of variables, and then try a few settings above and below that value. Triple bam! Hooray! We've made it to the end of another exciting StatQuest. Tune in next week and we'll talk about how to deal with missing data and how to cluster the samples. All right. Until then, quest on!

Frequently asked questions

Why are Decision Trees "not that awesome" in practice?

They work great on the data used to create them, but they lack flexibility and are inaccurate when classifying new, unseen samples.

What is the key difference between bootstrapping and standard sampling?

Bootstrapping involves sampling with replacement, meaning the same sample can be selected multiple times to create the new dataset.

What is "bagging"?

Bagging is the combination of bootstrapping the data and then aggregating the results (like voting) from the resulting models to make a final decision.

How many variables should I consider at each split?

A common starting point is the square root of the total number of variables; then test settings above and below this value.

What proportion of data usually ends up in the Out-of-Bag dataset?

Typically, about one-third (1/3) of the original data is not included in any specific bootstrapped dataset.