Standard Error & the CLT, In Plain English

Go back to Module 1's "Recap - Spotting Sampling Bias" lesson for a second. Buried in that Python cell is this line:

se = np.sqrt(p_hat * (1 - p_hat) / n)

You used it to show that a random sample of 5,000 gives a tighter margin of error than a random sample of 100. You've also used a close cousin of this exact formula in the CI lesson, the statsig-vs-shipsig lesson, and every A/B-testing tutorial in Module 3 - confint_proportions_2indep, proportions_ztest, and the napkin sample-size formula are all built on it. Nobody ever told you why that formula works, or why doubling precision costs 4x the sample. That's this lesson.

The one-paragraph intuition (the Central Limit Theorem)

Here's the single idea that makes every CI and z-test in this course legal to use: if you repeatedly draw random samples of a reasonable size (n ≳ 30) from any population - normal, skewed, bimodal, doesn't matter - and compute the mean (or proportion) of each sample, the distribution of those sample means will be approximately normal, centered on the true population value. This holds even when the individual data points are nothing like normal - a single user either converts or doesn't (a coin flip, not a bell curve), but the conversion rate across 10,000 users behaves like a bell curve. That's the Central Limit Theorem (CLT). It's the reason a z-test is a legitimate tool for a binary conversion metric at all.

Standard error is just the spread of that bell curve

Once you accept that sample proportions are approximately normally distributed, "standard error" (SE) is just the standard deviation of that distribution - how spread-out your estimate of p would be if you re-ran the same experiment over and over. For a proportion, that spread works out to:

SE = sqrt( p(1-p) / n )

which is exactly the line from Module 1. p(1-p) is the variance of a single binary observation (highest when p = 0.5, lowest near 0 or 1); dividing by n and taking the square root converts "spread of one observation" into "spread of the average of n observations."

Simulating the CLT: skewed inputs, normal output

You don't have to take the theorem on faith - you can watch it happen. Draw repeated samples from a population that's nowhere close to normal (a long-tailed LTV distribution, same shape as Module 0's "Spotting the Shape" lesson) and look at the distribution of the sample means:

import numpy as np
import matplotlib.pyplot as plt

# A population that is NOT normal at all: heavily right-skewed LTV
np.random.seed(42)
population = np.random.lognormal(mean=4.5, sigma=1.2, size=200_000)
print(f"Population shape - mean: {population.mean():.1f}, is it symmetric? No - heavily right-skewed.")

# Draw 2,000 independent samples of n=50 each, record each sample's mean
sample_means = [np.random.choice(population, size=50, replace=False).mean() for _ in range(2000)]

print(f"Sample-mean distribution - mean: {np.mean(sample_means):.1f}, std (this IS the standard error): {np.std(sample_means):.2f}")
# Plot: population histogram is a long right tail. sample_means histogram is a clean bell curve.
plt.hist(sample_means, bins=40)
plt.title("Distribution of 2,000 sample means (n=50 each) - approximately normal, despite a skewed population")
# plt.show()

The individual population is skewed. The distribution of sample means is not - it's approximately normal, centered on the true population mean, with a spread equal to the standard error. That's the CLT, made visible.

Why CI width shrinks as 1/√n, not 1/n

Look at the SE formula again: n sits inside a square root. That single detail explains a pattern you've now seen three times in this course (the polling table in Module 1, the CI-width MCQ in Module 1, and the sample-size math in Module 3):

To cut your margin of error in half, you don't need 2x the data - you need 4x the data, because √4 = 2. This is why precise estimates get expensive fast, and why a company bragging "we surveyed 50,000 people!" is often not meaningfully more precise than one that surveyed 5,000 - going from 5,000 to 50,000 (10x the data) only tightens the CI by about √10 ≈ 3.2x, not 10x.

Takeaway

Every confidence interval and z-test in this course leans on the same two facts: (1) the CLT says sample means/proportions are approximately normal for reasonably large n, even when the underlying data isn't, and (2) standard error - the spread of that normal curve - shrinks proportional to 1/√n. You've been using both silently since Module 1. Now you can explain why they work, not just run the code.