Bayesian Inference — The Update Game

Stop hiding from Bayes. Prior + likelihood = posterior. The update rule that powers modern probabilistic modeling. FIND_VIDEO: search 'Bayesian inference prior posterior' — recommended channel: 3Blue1Brown / StatQuest. Aim for 11 min or under.

14 minutesVideo LessonPDF notes

Watch Bayesian Inference — The Update Game on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Conditional Probability Review — Conditional probability is calculated by dividing the joint count by the count of the given condition.
Scaling by Knowledge — Conditional probability is the probability of an event scaled by the knowledge we already have about that event.
Comparing Conditionals — Changing the given knowledge changes the scaling factor (denominator) and thus the resulting conditional probability.
Deriving Bayes' Theorem — Bayes' Theorem is derived algebraically by setting two equivalent expressions for the joint probability equal to each other.
Standard Formula — The standard formula relates P(A|B) to P(B|A) using the marginal probabilities P(A) and P(B).
Importance of Bayes — Bayes' Theorem is crucial because it allows calculation of probabilities even when full data is unavailable and estimates must be used.
Notation Comparison — The standard notation for conditional probability omits the known event, which can obscure the fact that both inverse conditionals refer to the same joint event.

PDF notes

Bottom line: You will learn how to derive Bayes' Theorem from the definition of conditional probability using simple algebra. You will be able to calculate a conditional probability even when the joint probability is unknown, using prior knowledge and scaling factors.

This lesson on Bayesian Inference — The Update Game is hands-on and example-driven. You will learn how to derive Bayes' Theorem from the definition of conditional probability using simple algebra. You will be able to calculate a conditional probability even when the joint probability is unknown, using prior knowledge and scaling factors.

What You'll Be Able To Do

Derive Bayes' Theorem using algebraic manipulation of conditional probability equations.
Calculate a conditional probability using Bayes' Theorem when only marginal and inverse conditional probabilities are known.
Explain the relationship between joint probability and conditional probability using the concept of scaling.
Compare the standard and the slightly redundant notations for conditional probability.
Discuss why Bayes' Theorem is crucial when working with incomplete or estimated data.
Identify the components of Bayes' Theorem (prior, likelihood, posterior, evidence).

Topics Covered in Bayesian Inference — The Update Game

Conditional Probability Review (0:30 - 1:30) — Conditional probability is calculated by dividing the joint count by the count of the given condition.
Scaling by Knowledge (1:30 - 2:45) — Conditional probability is the probability of an event scaled by the knowledge we already have about that event.
Comparing Conditionals (2:45 - 4:00) — Changing the given knowledge changes the scaling factor (denominator) and thus the resulting conditional probability.
Deriving Bayes' Theorem (4:00 - 6:00) — Bayes' Theorem is derived algebraically by setting two equivalent expressions for the joint probability equal to each other.
Standard Formula (6:00 - 7:00) — The standard formula relates P(A|B) to P(B|A) using the marginal probabilities P(A) and P(B).
Importance of Bayes (7:00 - 8:30) — Bayes' Theorem is crucial because it allows calculation of probabilities even when full data is unavailable and estimates must be used.
Notation Comparison (8:30 - 9:45) — The standard notation for conditional probability omits the known event, which can obscure the fact that both inverse conditionals refer to the same joint event.

SQL Cheat Sheet

Conditional Probability — Probability of event A happening given that event B already happened
- In practice: P(No Candy | Soda) = P(No Candy and Soda) / P(Soda)
Joint Probability — Probability that two events (A and B) both occur simultaneously
- In practice: P(No Candy and Soda) = P(No Candy | Soda) * P(Soda)
Bayes' Theorem — Relates two inverse conditional probabilities using marginal probabilities
- In practice: P(A|B) = [P(B|A) * P(A)] / P(B)
Unconditional Probability — Probability of an event without any prior knowledge or conditions
- In practice: P(Soda) = 7 / 14
Scaling — Adjusting probability based on existing knowledge or conditions
- In practice: P(A and B) = P(A | B) * P(B)

Comparison Table

Notation Type	Example Form	Clarity of Event
Redundant Notation	P(A and B	B)
Standard Notation	P(A	B)
Key Insight	Numerator is always P(A and B)	Highlights the single joint event.

Common Pitfalls

Mistake: Assuming Bayes is only useful when all data is available. Avoid: Use Bayes when data is incomplete or requires estimated probabilities.
Mistake: Confusing P(A|B) with P(B|A) in the formula derivation. Avoid: Clearly identify the event being scaled and the scaling factor (knowledge).
Mistake: Forgetting to divide by the unconditional probability P(B). Avoid: Remember P(B) acts as the normalizing denominator (evidence).

FAQs

Why use the slightly redundant notation for conditional probability? It makes it obvious that both inverse conditional probabilities refer to the same joint event (A and B), differing only by the given knowledge.
Why did the calculated probability change when using Bayes' Theorem with StatSquatch's guess? The result changed because the input probability (P(Soda)) was an estimate, not the true value derived from the full data set.
What is the 'big deal' about Bayes' Theorem if it's just algebra? It allows calculation of conditional probabilities when the joint probability is unknown, which is essential when dealing with large, incomplete, or estimated data sets.
What is the relationship between Bayes' Theorem and Bayesian statistics? Bayes' Theorem is the mathematical basis for Bayesian statistics, which is a broader philosophy on how statistical inference should be calculated using prior beliefs.

Common pitfalls

Mistake: Assuming Bayes is only useful when all data is available. Avoid: Use Bayes when data is incomplete or requires estimated probabilities.
Mistake: Confusing P(A|B) with P(B|A) in the formula derivation. Avoid: Clearly identify the event being scaled and the scaling factor (knowledge).
Mistake: Forgetting to divide by the unconditional probability P(B). Avoid: Remember P(B) acts as the normalizing denominator (evidence).

Bayes theorem versus StatSquatch. Bayes theorem wins. StatQuest. Hello. I'm Josh Starmer, and welcome to StatQuest. Today we're going to talk about Bayes theorem, and it's going to be clearly explained. Note: This StatQuest assumes that you are already familiar with conditional probability. If not, check out the Quest. That said, let's do a quick review. In the StatQuest on conditional probability, we took a trip to StatLand and asked everyone, represented by a colorful dot, if they loved candy and/or soda. These two people loved candy and soda. These four people only loved candy. These five people only loved soda, and these three people didn't like candy and they didn't like soda. Bam. Then we calculated the probabilities for each cell in the contingency table by dividing the counts by the total number of people in StatLand, 14. Bam. Then we determined the total number and probability of people who loved soda and did not love soda, and the total number and probability of people who loved candy and did not love candy. Bam. Then we calculated the conditional probability that someone in StatLand might not love candy but love soda, given that we already know that they love soda. We did this by dividing the five people that do not love candy but love soda by the seven people that love soda, and we got 0.71. Then, just for fun, we divided the numerator and the denominator by the total population of StatLand, 14. Doing this extra division did not change the result. We still got 0.71. However, now the numerator is the original unconditional probability that someone in StatLand does not love candy but loves soda. And the denominator is the unconditional probability that someone in StatLand loves soda. All in all, the probability that someone does not love candy but loves soda, given that we know that they love soda, is equal to the probability that someone does not love candy but loves soda, divided by the probability that someone in StatLand loves soda. So, one way to think about conditional probability is the probability that an event will happen (in this case, the probability that we meet someone who does not love candy but loves soda) scaled by the knowledge we already have about the event. In this case, we know that the person loves soda. Note: Because saying "someone who does not love candy and love soda, given that they love soda" is a little redundant, many people omit writing out the "and love soda" part and shorten the conditional probability statement to "someone who does not love candy given that they love soda." While this notation is standard, it makes it harder to see the relationship between the thing we want to calculate the probability for on the left and how we calculate it on the right. Using the slightly redundant notation for conditional probability makes it obvious that we want the probability that an event happens scaled by the knowledge we already have about the event. Personally, this slightly redundant notation helped me get a better understanding of Bayes theorem, and we'll talk about this more at the end of the StatQuest. Now let's see what happens when we change what we already know about the event. From knowing that they love soda to knowing that they do not love candy. Now we have the probability that an event happens (which, in this case, is the event that we meet someone who does not love candy but love soda) scaled by the knowledge we already have about the event. In this case, we already know that they do not love candy. Now we plug in the numbers and do the math and get 0.63. Now let's compare this conditional probability, where we already know that the person does not love candy, to the conditional probability we calculated before, where we already knew that the person loves soda. In both cases, we want to know the probability of the same event: meeting someone who does not love candy but loves soda. And that means, in both cases, the numerators are the same. However, since we have different knowledge in each case, we scale the probabilities of the events differently, and ultimately, we get different probabilities. Hey look, it's StatSquatch. StatSquatch is our friend in StatLand, and he always wants to make a bet. I bet you one dollar that you can't solve the conditional probabilities without knowing the probability of not loving candy and loving soda. In other words, if we don't know the value for the numerators, can we still solve for the conditional probabilities? Well, even if we don't know the probability that someone does not love candy but loves soda, we can multiply both sides of the top equation by the probability that someone loves soda. And these two terms on the right cancel out. And we are left with the probability that we meet someone that does not love candy but loves soda equal to this stuff on the left side. Likewise, we can multiply both sides of the equation on the bottom by the probability of not loving candy. And these two terms on the right side cancel out. And just like before, we end up with the probability of meeting someone who does not love candy but loves soda equal to this stuff on the left side. Now we have two things on the left side of the equal signs that are both equal to the probability that we will meet someone who does not love candy and loves soda. Now, remember that StatSquatch asked us to solve for this term and this term without this term. And because both equations are equal to the term we want to omit, both equations are equal to each other. So let's move this up a little bit and move this over here. Now, remember we want to solve for this term, and we want to solve for this term. We'll start with the term on the left. First, we divide both sides by the probability that someone loves soda. And the probability that someone loves soda cancels out on the left side, and we have solved for this term. Bam. Now let's move the thing on the right to the left and the thing on the left to the right and divide both sides by the probability that someone does not love candy. And the probability that someone does not love candy cancels out on the left side, and we have solved for the other term. Bam. In both cases, we won the bet with StatSquatch because we no longer need to know the probability that someone does not love candy and love soda. But more importantly, we have derived Bayes theorem. Double bam. Bayes theorem tells us that this conditional probability, which is based on knowing that the person loves soda, can be derived from this conditional probability, which is based on knowing that they do not love candy. Alternatively, Bayes's theorem tells us that this conditional probability, which is based on knowing that the person does not love candy, can be derived from this conditional probability based on knowing they love soda. In general, if we let A equal "does not love candy" and B equals "loves soda," then we can rewrite each equation into the standard formula for Bayes's theorem. In other words, the conditional probability given that we know one thing about an event can be derived from knowing the other thing about the event. Now StatSquatch says, "Dude, you derived Bayes theorem with just a little algebra. What's the big deal?" When we have all of the data laid out in a nice colorful chart or in a contingency table, then Bayes theorem is not that big of a deal. In fact, when you have all of the data, Bayes theorem isn't even a small deal. However, most of the time we don't have all of the data. In other words, StatSquatch might only tell us, "The probability that someone does not love candy, given that they love soda is 0.71, and I'm not certain, but I think the probability that someone loves soda is close to 0.6. And the probability that someone does not like candy is 0.57." And if this is all the data we have, then we plug the numbers into Bayes theorem and get approximately 0.75. And that means, given this data, which includes a guess about the probability someone loves soda, the probability that someone loves soda, given that we know they do not like candy, is about 0.75. Note: Attentive viewers may notice that when we calculated the conditional probability with Bayes theorem, which we used because we did not have all of the information, the result is different from when we calculated the probability knowing everything. This is because StatSquatch didn't know the exact value for the probability that someone loves soda. They just took a guess. And, while taking a guess might sound like a terrible thing to do, it's the only option when we have a large population. For example, it would be almost impossible to ask every single person in India if they love soda. So, a lot of times, we have to make a guess. Bayesian statistics is about understanding what it means to make a guess like this and all it implies. Bayes theorem is the basis for Bayesian statistics, which is this equation paired with a broader philosophy of how statistics should be calculated. And we'll cover all these topics in follow-up StatQuests. However, before we go, I want to review the standard notation so if you research Bayes theorem or Bayesian statistics on your own, you won't be totally lost. Like I said earlier, when most people write conditional probabilities, they do it differently from the examples I've given here. Specifically, because they know that this person does not love candy, they do not include it when stating the probability. Now the conditional probability reads, "The probability someone loves soda given that they do not love candy." Likewise, because they know this person loves soda, they do not include it when stating the probability. Now the conditional probability reads, "The probability that someone does not love candy given that they love soda." In the end, it looks like we want to calculate the probability of two different events. However, it is important to keep in mind that in both cases, there is only one event, and both conditional probabilities refer to the same yellow area in the drawing and the same yellow square in the contingency table. And the only real difference between the two conditional probabilities is the given knowledge. And that's why I prefer the longer, slightly redundant way to write conditional probabilities, because the longer way makes it obvious that in both cases, we are talking about the exact same thing. Small bam? No. Triple bam. Now it's time for some shameless self-promotion. If you want to review statistics and machine learning offline, check out the StatQuest study guides at statquest.org. There's something for everyone. Hooray! We've made it to the end of another exciting StatQuest. If you liked this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a T-shirt or a hoodie, or just donate. The links are in the description below. All right, until next time, quest on.

Frequently asked questions

Why use the slightly redundant notation for conditional probability?

It makes it obvious that both inverse conditional probabilities refer to the same joint event (A and B), differing only by the given knowledge.

Why did the calculated probability change when using Bayes' Theorem with StatSquatch's guess?

The result changed because the input probability (P(Soda)) was an estimate, not the true value derived from the full data set.

What is the 'big deal' about Bayes' Theorem if it's just algebra?

It allows calculation of conditional probabilities when the joint probability is unknown, which is essential when dealing with large, incomplete, or estimated data sets.

What is the relationship between Bayes' Theorem and Bayesian statistics?

Bayes' Theorem is the mathematical basis for Bayesian statistics, which is a broader philosophy on how statistical inference should be calculated using prior beliefs.