Spotting Sampling Bias

# Spotting Sampling Bias There's a famous result in stats: in 1936, *Literary Digest* polled 2.4 million Americans about the upcoming presidential election. They predicted Landon would beat Roosevelt in a landslide. Roosevelt won by the largest margin in modern history. A nobody named George Gallup polled 50,000 people and called the result correctly. How? The Digest polled phone owners and magazine subscribers - wealthy Americans, who skewed Republican. Gallup randomly sampled the population. Sample size didn't beat sample design. It almost never does. ## Sampling precision & standard error in Python ```python import numpy as np import pandas as pd # Population: 100,000 customers with true satisfaction rate of 72% (0.72) np.random.seed(42) population_sat = np.random.binomial(n=1, p=0.72, size=100000) df_pop = pd.DataFrame({"satisfied": population_sat}) # Draw independent random samples of different sizes sample_sizes = [100, 400, 1000, 5000] results = [] for n in sample_sizes: sample = df_pop["satisfied"].sample(n=n, random_state=42) p_hat = sample.mean() # Standard error and 95% Margin of Error (z = 1.96) se = np.sqrt(p_hat * (1 - p_hat) / n) moe = 1.96 * se results.append({ "Sample Size (N)": n, "Estimated p": f"{p_hat:.3f}", "Standard Error": f"{se:.4f}", "Margin of Error (±95%)": f"±{moe:.2%}", "95% CI Range": f"[{p_hat - moe:.1%}, {p_hat + moe:.1%}]" }) summary_df = pd.DataFrame(results) print(summary_df.to_string(index=False)) ``` ## Sample size: the napkin version For polling-style questions (yes/no, conversion rates): | Sample size | Margin of error | |---|---| | 100 | ±10% | | 400 | ±5% | | 1,000 | ±3% | | 4,000 | ±1.5% | Doubling precision requires 4× the sample size. There's massive diminishing returns above ~1,000. If someone tells you "we polled 50,000 users", the marginal precision over 1,000 is tiny - what matters is who those 50,000 are. ## Three biases that bite analysts daily **Selection bias** - your sample isn't randomly drawn. The people who answer your NPS survey are not your average user; they're either delighted or furious. NPS scores systematically misrepresent your middle-of-the-road majority. **Survivorship bias** - you only see the data of the things that survived. "Successful startups always have X" - but the failed ones also had X. You're only looking at survivors. **Self-selection bias** - your sample chose to be in the sample. Anyone who fills out a 20-question feedback form is, by definition, more engaged than your average user. ## Takeaway Random > large. A small random sample beats a huge biased one almost every time. When you read "we surveyed 10,000 customers", the next sentence in your head should be "out of how many, and who responded?"