Spotting Sampling Bias

There's a famous result in stats: in 1936, Literary Digest polled 2.4 million Americans about the upcoming presidential election. They predicted Landon would beat Roosevelt in a landslide. Roosevelt won by the largest margin in modern history.

A nobody named George Gallup polled 50,000 people and called the result correctly.

How? The Digest polled phone owners and magazine subscribers - wealthy Americans, who skewed Republican. Gallup randomly sampled the population. Sample size didn't beat sample design. It almost never does.

Sampling precision & standard error in Python

import numpy as np
import pandas as pd

# Population: 100,000 customers with true satisfaction rate of 72% (0.72)
np.random.seed(42)
population_sat = np.random.binomial(n=1, p=0.72, size=100000)
df_pop = pd.DataFrame({"satisfied": population_sat})

# Draw independent random samples of different sizes
sample_sizes = [100, 400, 1000, 5000]
results = []

for n in sample_sizes:
    sample = df_pop["satisfied"].sample(n=n, random_state=42)
    p_hat = sample.mean()
    # Standard error and 95% Margin of Error (z = 1.96)
    se = np.sqrt(p_hat * (1 - p_hat) / n)
    moe = 1.96 * se
    results.append({
        "Sample Size (N)": n,
        "Estimated p": f"{p_hat:.3f}",
        "Standard Error": f"{se:.4f}",
        "Margin of Error (±95%)": f"±{moe:.2%}",
        "95% CI Range": f"[{p_hat - moe:.1%}, {p_hat + moe:.1%}]"
    })

summary_df = pd.DataFrame(results)
print(summary_df.to_string(index=False))

Sample size: the napkin version

For polling-style questions (yes/no, conversion rates):

Doubling precision requires 4× the sample size. There's massive diminishing returns above ~1,000. If someone tells you "we polled 50,000 users", the marginal precision over 1,000 is tiny - what matters is who those 50,000 are.

Three biases that bite analysts daily

Selection bias - your sample isn't randomly drawn. The people who answer your NPS survey are not your average user; they're either delighted or furious. NPS scores systematically misrepresent your middle-of-the-road majority.

Survivorship bias - you only see the data of the things that survived. "Successful startups always have X" - but the failed ones also had X. You're only looking at survivors.

Self-selection bias - your sample chose to be in the sample. Anyone who fills out a 20-question feedback form is, by definition, more engaged than your average user.

Takeaway

Random > large. A small random sample beats a huge biased one almost every time. When you read "we surveyed 10,000 customers", the next sentence in your head should be "out of how many, and who responded?"