Standard Error & the CLT, In Plain English

# Standard Error & the CLT, In Plain English Go back to Module 1's "Recap - Spotting Sampling Bias" lesson for a second. Buried in that Python cell is this line: ```python se = np.sqrt(p_hat * (1 - p_hat) / n) ``` You used it to show that a random sample of 5,000 gives a tighter margin of error than a random sample of 100. You've also used a close cousin of this exact formula in the CI lesson, the statsig-vs-shipsig lesson, and every A/B-testing tutorial in Module 3 - `confint_proportions_2indep`, `proportions_ztest`, and the napkin sample-size formula are all built on it. Nobody ever told you *why* that formula works, or why doubling precision costs 4x the sample. That's this lesson. ## The one-paragraph intuition (the Central Limit Theorem) Here's the single idea that makes every CI and z-test in this course legal to use: if you repeatedly draw random samples of a reasonable size (n ≳ 30) from *any* population - normal, skewed, bimodal, doesn't matter - and compute the mean (or proportion) of each sample, the distribution of those sample means will be **approximately normal**, centered on the true population value. This holds even when the individual data points are nothing like normal - a single user either converts or doesn't (a coin flip, not a bell curve), but the *conversion rate across 10,000 users* behaves like a bell curve. That's the Central Limit Theorem (CLT). It's the reason a z-test is a legitimate tool for a binary conversion metric at all. ## Standard error is just the spread of that bell curve Once you accept that sample proportions are approximately normally distributed, "standard error" (SE) is just the standard deviation *of that distribution* - how spread-out your estimate of `p` would be if you re-ran the same experiment over and over. For a proportion, that spread works out to: ``` SE = sqrt( p(1-p) / n ) ``` which is exactly the line from Module 1. `p(1-p)` is the variance of a single binary observation (highest when `p = 0.5`, lowest near 0 or 1); dividing by `n` and taking the square root converts "spread of one observation" into "spread of the average of n observations." ## Simulating the CLT: skewed inputs, normal output You don't have to take the theorem on faith - you can watch it happen. Draw repeated samples from a population that's nowhere close to normal (a long-tailed LTV distribution, same shape as Module 0's "Spotting the Shape" lesson) and look at the distribution of the *sample means*: ```python import numpy as np import matplotlib.pyplot as plt # A population that is NOT normal at all: heavily right-skewed LTV np.random.seed(42) population = np.random.lognormal(mean=4.5, sigma=1.2, size=200_000) print(f"Population shape - mean: {population.mean():.1f}, is it symmetric? No - heavily right-skewed.") # Draw 2,000 independent samples of n=50 each, record each sample's mean sample_means = [np.random.choice(population, size=50, replace=False).mean() for _ in range(2000)] print(f"Sample-mean distribution - mean: {np.mean(sample_means):.1f}, std (this IS the standard error): {np.std(sample_means):.2f}") # Plot: population histogram is a long right tail. sample_means histogram is a clean bell curve. plt.hist(sample_means, bins=40) plt.title("Distribution of 2,000 sample means (n=50 each) - approximately normal, despite a skewed population") # plt.show() ``` The individual population is skewed. The distribution of sample *means* is not - it's approximately normal, centered on the true population mean, with a spread equal to the standard error. That's the CLT, made visible. ## Why CI width shrinks as 1/√n, not 1/n Look at the SE formula again: `n` sits *inside a square root*. That single detail explains a pattern you've now seen three times in this course (the polling table in Module 1, the CI-width MCQ in Module 1, and the sample-size math in Module 3): | Sample size | Standard error (relative) | |---|---| | n | 1x | | 4n | 1/2x | | 16n | 1/4x | | 100n | 1/10x | To cut your margin of error in half, you don't need 2x the data - you need **4x** the data, because `√4 = 2`. This is why precise estimates get expensive fast, and why a company bragging "we surveyed 50,000 people!" is often not meaningfully more precise than one that surveyed 5,000 - going from 5,000 to 50,000 (10x the data) only tightens the CI by about √10 ≈ 3.2x, not 10x. ## Takeaway Every confidence interval and z-test in this course leans on the same two facts: (1) the CLT says sample means/proportions are approximately normal for reasonably large n, even when the underlying data isn't, and (2) standard error - the spread of that normal curve - shrinks proportional to 1/√n. You've been using both silently since Module 1. Now you can explain *why* they work, not just run the code.