Back to Evaluation, Leakage, and Unsupervised

Evaluation Metrics — AUC, F1, Precision/Recall, Brier

No metric is universally right. Pick by what the model will be used for. FIND_VIDEO: search 'ROC AUC F1 precision recall explained' — recommended channel: StatQuest. Aim for 11 min or under.

16 minutesVideo LessonPDF notes

Watch Evaluation Metrics — AUC, F1, Precision/Recall, Brier on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Setup and Context — The lesson builds on confusion matrices and logistic regression basics.
Classification Threshold — A logistic regression curve assigns probabilities, requiring a threshold (e.g., 0.5) for classification.
Evaluating 0.5 Threshold — Testing the 0.5 threshold generates a confusion matrix used to calculate sensitivity and specificity.
Variable Thresholds — Changing the threshold alters the balance between false positives and false negatives, depending on the application cost.
ROC Axes Defined — The ROC graph plots True Positive Rate (Sensitivity) on the Y-axis against False Positive Rate (1 - Specificity) on the X-axis.
Plotting the Curve — Varying the classification threshold generates a series of (FPR, TPR) points that form the ROC curve.
Interpreting ROC — Points further to the upper-left indicate better performance, helping identify the optimal threshold.
Area Under Curve (AUC) — AUC summarizes the overall model performance, allowing easy comparison between different classification methods.
Precision and Imbalance — Precision is introduced as an alternative metric that is less affected by highly imbalanced datasets because it excludes true negatives.

PDF notes

Bottom line: You will be able to evaluate binary classification models using metrics derived from the confusion matrix. You will learn how to use the ROC curve and AUC to select the optimal classification threshold and compare different models effectively.

This lesson on Evaluation Metrics — AUC, F1, Precision/Recall, Brier is hands-on and example-driven. You will be able to evaluate binary classification models using metrics derived from the confusion matrix. You will learn how to use the ROC curve and AUC to select the optimal classification threshold and compare different models effectively.

What You'll Be Able To Do

Calculate True Positive Rate (Sensitivity) and False Positive Rate (1 - Specificity).
Interpret the meaning of points on a Receiver Operator Characteristic (ROC) curve.
Determine the optimal classification threshold based on the ROC curve position.
Compare the performance of two classification models using the Area Under the Curve (AUC).
Explain why Precision might be preferred over False Positive Rate in imbalanced datasets.

Topics Covered in Evaluation Metrics — AUC, F1, Precision/Recall, Brier

Setup and Context (0:00 - 1:00) — The lesson builds on confusion matrices and logistic regression basics.
Classification Threshold (1:00 - 2:30) — A logistic regression curve assigns probabilities, requiring a threshold (e.g., 0.5) for classification.
Evaluating 0.5 Threshold (2:30 - 4:00) — Testing the 0.5 threshold generates a confusion matrix used to calculate sensitivity and specificity.
Variable Thresholds (4:00 - 6:00) — Changing the threshold alters the balance between false positives and false negatives, depending on the application cost.
ROC Axes Defined (6:00 - 8:00) — The ROC graph plots True Positive Rate (Sensitivity) on the Y-axis against False Positive Rate (1 - Specificity) on the X-axis.
Plotting the Curve (8:00 - 11:00) — Varying the classification threshold generates a series of (FPR, TPR) points that form the ROC curve.
Interpreting ROC (11:00 - 12:00) — Points further to the upper-left indicate better performance, helping identify the optimal threshold.
Area Under Curve (AUC) (12:00 - 13:00) — AUC summarizes the overall model performance, allowing easy comparison between different classification methods.
Precision and Imbalance (13:00 - 14:00) — Precision is introduced as an alternative metric that is less affected by highly imbalanced datasets because it excludes true negatives.

SQL Cheat Sheet

ROC Curve — Summarizes confusion matrices across all possible thresholds
```
SELECT TPR, FPR FROM model_thresholds ORDER BY threshold DESC;
```

AUC — Area under the ROC curve; measures overall model performance

SELECT model_name, AUC_score FROM model_comparison WHERE AUC_score > 0.8;

True Positive Rate (TPR) — Proportion of actual positive samples correctly classified (Sensitivity)
```
SELECT TP / (TP + FN) AS TPR FROM confusion_matrix;
```
False Positive Rate (FPR) — Proportion of actual negative samples incorrectly classified
```
SELECT FP / (FP + TN) AS FPR FROM confusion_matrix;
```
Precision — Proportion of positive predictions that were actually correct
```
SELECT TP / (TP + FP) AS Precision FROM confusion_matrix;
```

Comparison Table

Metric	Formula	Purpose
TPR (Sensitivity)	TP / (TP + FN)	Proportion of actual positives found.
FPR (1 - Specificity)	FP / (FP + TN)	Proportion of actual negatives missed.
Precision	TP / (TP + FP)	Accuracy of positive predictions.

Common Pitfalls

Mistake: Assuming 0.5 is always the best threshold. Avoid: Use the ROC curve to find the threshold balancing costs.
Mistake: Confusing TPR (Sensitivity) with Precision. Avoid: TPR uses actual positives; Precision uses predicted positives.
Mistake: Relying only on FPR for imbalanced data. Avoid: Use Precision/Recall when true negatives heavily outweigh positives.
Mistake: Interpreting AUC as a percentage accuracy. Avoid: AUC is the probability a positive sample ranks higher than a negative.

FAQs

Why use ROC/AUC instead of just accuracy? ROC/AUC evaluates performance across all possible thresholds, providing a complete picture independent of a single cutoff.
What does a point on the diagonal line (TPR=FPR) mean? The model performs no better than random guessing at that specific threshold.
When is Precision more useful than FPR? When the dataset is highly imbalanced (e.g., rare disease), Precision is less affected by the large number of true negatives.
What does a perfect ROC curve look like? A perfect curve goes straight up from (0, 0) to (0, 1) and then straight across to (1, 1), resulting in an AUC of 1.0.

📊 PR-AUC vs ROC-AUC on Imbalanced Data (+ Brier Calibration)

Rule of thumb: on balanced data ROC-AUC and PR-AUC usually agree; on imbalanced data (fraud, churn, CTR) trust PR-AUC. ROC-AUC counts easy true negatives, so a model that predicts "all negative" still scores ~0.5 ROC-AUC while PR-AUC collapses to the prevalence baseline.

Random baseline: ROC-AUC = 0.5 always; PR-AUC = positive rate (e.g. 0.003 at 0.3% fraud). Judge PR-AUC against prevalence, not against 1.0.
When probabilities become money (bidding, credit limits, triage thresholds), ranking is not enough — check calibration with reliability plots and the Brier score (mean squared error of P(y) vs outcome).
Interview line: "I report PR-AUC plus precision/recall at the deploy threshold, and Brier whenever the score is consumed as a probability."

Common pitfalls

Mistake: Assuming 0.5 is always the best threshold. Avoid: Use the ROC curve to find the threshold balancing costs.
Mistake: Confusing TPR (Sensitivity) with Precision. Avoid: TPR uses actual positives; Precision uses predicted positives.
Mistake: Relying only on FPR for imbalanced data. Avoid: Use Precision/Recall when true negatives heavily outweigh positives.
Mistake: Interpreting AUC as a percentage accuracy. Avoid: AUC is the probability a positive sample ranks higher than a negative.

Wait 'til you see The ROC And AUC They're cool, yeah, StatQuest. Hello, I'm Josh Starmer, and welcome to StatQuest. Today we're going to talk about ROC and AUC, and they're going to be clearly explained. Note: This StatQuest builds on the confusion matrix and sensitivity and specificity StatQuests. So, if you're not already down with those, check out the Quests. Also, the example I give in this StatQuest is based on logistic regression. So, even though ROC and AUC apply to more than just logistic regression, make sure you understand those basics. Let's start with some data. The Y axis has two categories: obese and not obese. The blue dots represent obese mice. And the red dots represent mice that are not obese. Along the X axis we have weight. This mouse is not obese even though it weighs a lot. It must be mighty mouse and just full of muscles. This mouse doesn't weigh that much, but it is still considered obese for its size. Now let's fit a logistic regression curve to the data. When we're doing logistic regression, the Y axis is converted to the probability that a mouse is obese. Now, let's just look at the curve. If someone told us that they had a heavy mouse that weighs this much, then the curve would tell us that there is a high probability that the mouse is obese. If someone told us that they had a light mouse that weighs this much, then the curve would tell us that there is a low probability that the mouse is obese. So, this logistic regression tells us the probability that a mouse is obese based on its weight. However, if we want to classify the mice as obese or not obese, then we need a way to turn probabilities into classifications. One way to classify mice is to set a threshold at 0.5. And classify all mice with a probability of being obese greater than 0.5 as obese, and classify all mice with a probability of being obese less than or equal to 0.5 as not obese. Using 0.5 as the cutoff, we would call this mouse obese. And this mouse not obese. If another mouse weighed this much, then we would classify it as obese. And if another mouse weighed this much, then we would classify it as not obese. To evaluate the effectiveness of this logistic regression with the classification threshold set to 0.5, we can test it with mice that we know are obese and not obese. Here are the weights of four new mice that we know are not obese. And here are the weights of four new mice that we know are obese. We know that this mouse is not obese, and the logistic regression with the classification threshold set to 0.5 correctly classifies it as not obese. This mouse is also correctly classified. But this mouse is incorrectly classified. We know that it is obese, but it is classified as not obese. The next mouse is correctly classified. But this mouse is incorrectly classified. The last three mice are correctly classified. Now we create a confusion matrix to summarize the classifications. These three samples were correctly classified as obese. And this sample was predicted to be obese, but was not obese. These three samples were correctly classified as not obese. And this sample was predicted to be not obese, even though it was obese. Once the confusion matrix is filled in, we can calculate sensitivity and specificity to evaluate this logistic regression when 0.5 is the threshold for obesity. Little bam, because so far, this is all review. Now let's talk about what happens when we use a different threshold for deciding if a sample is obese or not. For example, if it was super important to correctly classify every obese sample, we could set the threshold to 0.1. This would result in correct classifications for all four obese mice. But it would also increase the number of false positives. The lower threshold would also reduce the number of false negatives because all of the obese mice were correctly classified. Note: If the idea of using a threshold other than 0.5 is blowing your mind, imagine that instead of classifying samples as obese or not obese, we were classifying samples as infected with Ebola and not infected with Ebola. In this case, it's absolutely essential to correctly classify every sample infected with Ebola in order to minimize the risk of an outbreak. And that means lowering the threshold, even if that results in more false positives. On the other hand, we could set the threshold to 0.9. In this case, we would correctly classify the same number of obese samples as when the threshold was set to 0.5. But we wouldn't have any false positives. And we would correctly classify one more sample that was not obese. And have the same number of false negatives as before. With this data, the higher threshold does a better job classifying samples as obese or not obese. But the threshold could be set to anything between 0 and 1. How do we determine which threshold is the best? For starters, we don't need to test every single option. For example, these thresholds result in the exact same confusion matrix. But even if we made one confusion matrix for each threshold that mattered, it would result in a confusingly large number of confusion matrices. So, instead of being overwhelmed with confusion matrices, receiver operator characteristic (ROC) graphs provide a simple way to summarize all of the information. The Y axis shows the true positive rate, which is the same thing as sensitivity. The true positive rate is the true positives divided by the sum of the true positives and the false negatives. In this example, the true positives are the samples that were correctly classified as obese. And the false negatives are the obese samples that were incorrectly classified as not obese. The true positive rate tells you what proportion of obese samples were correctly classified. The X axis shows the false positive rate, which is the same thing as 1 minus specificity. The false positive rate is the false positives divided by the sum of the false positives and true negatives. The false positives are the non-obese samples that were incorrectly classified as obese. And the true negatives are the samples correctly classified as not obese. The false positive rate tells you the proportion of not obese samples that were incorrectly classified and are false positives. To get a better sense of how the ROC works, let's draw one from start to finish using our example data. We'll start by using a threshold that classifies all of the samples as obese. And that gives us this confusion matrix. First, let's calculate the true positive rate. There are four true positives. And there were zero false negatives. Doing the math gives us 1. The true positive rate when the threshold is so low that every single sample is classified as obese is 1. This means that every single obese sample was correctly classified. Now let's calculate the false positive rate. There were four false positives in the confusion matrix. And there were zero true negatives. Doing the math gives us 1. The false positive rate when the threshold is so low that every single sample is classified as obese is also 1. This means that every single sample that was not obese was incorrectly classified as obese. Now plot a point at (1, 1). A point at (1, 1) means that even though we correctly classified all of the obese samples, we incorrectly classified all of the samples that were not obese. This green diagonal line shows where the true positive rate equals the false positive rate. Any point on this line means that the proportion of correctly classified obese samples is the same as the proportion of incorrectly classified samples that are not obese. Going back to the logistic regression, let's increase the threshold so that all but the lightest sample are called obese. The new threshold gives us this confusion matrix. We then calculate the true positive rate and the false positive rate. And plot a point at (0.75, 1). Since the new point is to the left of the dotted green line, we know that the proportion of correctly classified samples that were obese is greater than the proportion of samples that were incorrectly classified as obese. In other words, the new threshold for deciding if a sample is obese or not is better than the first one. Now, let's increase the threshold so that all but the two lightest samples are called obese. The new threshold gives us this confusion matrix. We then calculate the true positive rate and the false positive rate. And plot a point at (0.5, 1). The new point is even further to the left of the dotted green line, showing that the new threshold further decreases the proportion of samples that were incorrectly classified as obese. In other words, the new threshold is the best one so far. Now we increase the threshold again. Create a confusion matrix. Calculate the true positive rate and the false positive rate. And plot the point. Now we increase the threshold again. Create a confusion matrix. Calculate the true positive rate and the false positive rate. And plot the point. The threshold represented by the new point correctly classifies 75% of the obese samples and 100% of the samples that were not obese. In other words, this threshold resulted in no false positives. Now we increase the threshold again. And plot the point. Now we increase the threshold again. And plot the point. Lastly, we choose a threshold that classifies all of the samples as not obese. And plot the point. The point at (0, 0) represents a threshold that results in zero true positives and zero false positives. If we want, we can connect the dots. And that gives us an ROC graph. The ROC graph summarizes all of the confusion matrices that each threshold produced. Without having to sort through the confusion matrices, I can tell that this threshold is better than this threshold. And depending on how many false positives I'm willing to accept, the optimal threshold is either this one or this one. Bam! Now that we know what an ROC graph is, let's talk about the area under the curve, or AUC. The AUC is 0.9. Bam! The AUC makes it easy to compare one ROC curve to another. The AUC for the red ROC curve is greater than the AUC for the blue ROC curve, suggesting that the red curve is better. So, if the red ROC curve represented logistic regression and the blue ROC curve represented a random forest, you would use the logistic regression. Double bam! Now, one last thing before we're all done. Although ROC graphs are drawn using true positive rates and false positive rates to summarize confusion matrices, there are other metrics that attempt to do the same thing. For example, people often replace the false positive rate with precision. Precision is the true positives divided by the sum of the true positives and false positives. Precision is the proportion of positive results that were correctly classified. If there were lots of samples that were not obese relative to the number of obese samples, then precision might be more useful than the false positive rate. This is because precision does not include the number of true negatives in its calculation and is not affected by the imbalance. In practice, this sort of imbalance occurs when studying a rare disease. In this case, the study will contain many more people without the disease than with the disease. Bam! In summary, ROC curves make it easy to identify the best threshold for making a decision. This threshold is better than this one. And the AUC can help you decide which categorization method is better. The red method is better than the blue method. Hooray! We've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, well, consider getting a T-shirt or a hoodie or buying one or two of my original songs. The links for doing that are below. All right, until next time, quest on!

Frequently asked questions

Why use ROC/AUC instead of just accuracy?

ROC/AUC evaluates performance across all possible thresholds, providing a complete picture independent of a single cutoff.

What does a point on the diagonal line (TPR=FPR) mean?

The model performs no better than random guessing at that specific threshold.

When is Precision more useful than FPR?

When the dataset is highly imbalanced (e.g., rare disease), Precision is less affected by the large number of true negatives.

What does a perfect ROC curve look like?

A perfect curve goes straight up from (0, 0) to (0, 1) and then straight across to (1, 1), resulting in an AUC of 1.0.