Bootstrap — Inference Without Distributional Assumptions

Bootstrapping StatQuest Hello, I'm Josh Starmer and welcome to StatQuest. Today we're going to talk about bootstrapping part one main ideas. Now, imagine we had a new drug to treat an illness. And we gave that drug to eight different people that had the illness. For five of these people, the drug appeared to help them feel better. But for three people, the drug appeared to make them feel worse. If we calculate the mean of the response to the drug, we get 0.5. 0.5 is not a huge improvement, but since most of the people, five of eight, improved, maybe this drug is better than using no drug at all. However, maybe these five people all felt better because they were healthier to begin with. And maybe these three people all felt worse because they had unhealthy lifestyles. So it is possible that the reason we got a mean value equal to 0.5 instead of zero, is because of random things that we can't control. Is there anything we can do to decide if the drug works or not? Yes. One expensive and time-consuming option would be to replicate the experiment a bunch of times. If we repeat the experiment a bunch of times, then we can keep track of each mean value. And we will end up with a histogram of mean values. Just by looking at this distribution, we can see that mean values close to zero, which suggests that the drug does not do anything, are relatively likely to occur. And mean values far from zero, indicating that the drug does something, are relatively rare. However, as I said earlier, repeating the experiment a bunch of times is both expensive and time-consuming. Is there something else we can do that is less expensive and time-consuming? Yes. Instead of replicating the experiment a bunch of times, we can use bootstrapping. Bam! So let's use bootstrapping to get a better sense of which results are likely and which are rare. First, let's create a new number line. Now, from the eight original measurements, choose one at random, and add that value to the new number line. Now go back to the original eight measurements and choose another value at random, and add it to the new number line. Then we repeat that process, randomly selecting one of the eight original values for the new number line a total of eight times. Note, we can randomly select the same value more than once. Oh no, it's the dreaded terminology alert. Randomly selecting data and allowing for duplicates is called sampling with replacement. Anyway, so far we've only selected six measurements, so we need two more. Note, the reason we selected eight measurements for the new number line is because the original data set that we are sampling from contains eight measurements. If we had started with 10 measurements, then we would need to add 10 measurements to the new number line. Anyway, this new data set that was created using sampling with replacement so that it had the same number of values as the original data set, is called a bootstrapped data set. Okay, now that we have a new bootstrapped data set, we calculate the mean. Note, because the bootstrapped data set is different from the original data set, we get a different mean. Now let's add the mean of the bootstrapped data set to what will soon be a histogram of means. Now we start over with a fresh number line, and randomly select from the eight original values for the new number line, repeating a total of eight times and allowing duplications. Then we calculate the mean, and add that to our histogram. Note, this process of creating a bootstrap data set, then calculating something, in this case, we calculate the mean, then keeping track of those calculations is called bootstrapping. In other words, bootstrapping consists of four steps. First, make a bootstrapped data set. Second, calculate something, in this case, we calculated the mean. Three, keep track of that calculation. And four, repeat steps one through three a bunch of times. Note, in step two, we calculated the mean, but we could have just as easily calculated the median or the standard deviation or any other statistic. Later on, I'll say more about why this flexibility is awesome. For now, I'll just say, bam! Okay, now that we know what bootstrapping is, we just do it a bunch of times. Usually, we use a computer to bootstrap thousands of times. And after creating thousands of bootstrapped samples, calculating their means, and adding them to the histogram, we end up with this. Because we sampled with replacement, the histogram ended up with a wide variety of mean values. Because there are so many combinations, bootstrapping usually only creates a subset, like 10,000, to estimate the full distribution. In this case, the histogram tells us how the mean might change if we redid the experiment a bunch of times. Just by looking at the histogram, we can get a sense of what might happen if we redid the experiment. If we redid the experiment, there's a high likelihood we will get something close to the original mean. And getting something really far from the original mean should be relatively rare. Because the histogram tells us how the mean might change if we redid the experiment a bunch of times, if we want to know the standard error of the mean value from the original data set, we only need to calculate the standard deviation of this distribution. And a 95% confidence interval is just an interval that covers 95% of the bootstrapped means. Double bam! In this case, we see that the 95% confidence interval covers zero. So we cannot reject the hypothesis that the drug is not doing anything. Note, what we just did with the confidence interval was a type of hypothesis testing. If you want to learn more about hypothesis testing, check out the Quest. Also note, just so you know, there are other, fancier ways to use bootstrapping to calculate confidence intervals. While these fancy methods can result in better confidence intervals, we'll save them for another day, since the purpose of this video is to explain the main ideas behind bootstrapping. Small bam. Now, so far we have used bootstrapping to calculate the standard error and a confidence interval for the mean. However, both the standard error and the confidence interval can be calculated directly with a formula, without having to create bootstrapped data sets. So what is it that makes bootstrapping so awesome? The awesome thing about bootstrapping is that we can apply it to any statistic to create a histogram of what might happen if we repeated the experiment a bunch of times. And we can use that histogram to calculate stuff like standard errors and confidence intervals without having to worry about whether or not there is a nice formula. For example, if we started out by calculating the median of the original data, then we can use bootstrapping to create a distribution, and use that distribution to create the confidence interval. So, regardless of the statistic we calculate, bootstrapping allows us to see it in the context of a distribution. And we can use that distribution to help us interpret the initial results. Triple bam! Now it's time for some shameless self-promotion. If you want to review statistics and machine learning offline, check out the StatQuest Study Guides at statquest.org. There's something for everyone. Hooray! We've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a t-shirt or a hoodie, or just donate. The links are in the description below. All right, until next time, quest on!