EDA in Python: A Checklist That Catches Silent Data Errors

Every messy dataset breaks the same way — mixed types, silent nulls, and a histogram you skipped. This playbook gives you a 10-step EDA template that plugs into any CSV, built from the synthetic generator in courses/workbooks/generators/linkedinEDA.py.

What does the EDA playbook cover and who is it for?

This notebook is a template, not a one-off report. It walks from environment setup through data loading, initial exploration, cleaning, univariate and bivariate analysis, correlation, and feature engineering — the exact order hiring managers expect in a take-home. Use it as your default first pass whenever you open a new dataset, then jump to the Pandas cheatsheet for one-liner patterns and SQL NULL handling when missing values come from a database.

Ingredients: a 1,000-row, 20-feature synthetic classification dataset via sklearn.datasets.make_classification (10 informative, 5 redundant) plus engineered nulls and outliers so every checklist step has something to catch.

How do you set up and load data correctly?

Start every EDA with pinned configuration so outputs are reproducible.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
 
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 100)
sns.set_theme(style='whitegrid')
warnings.filterwarnings('ignore')

Load the synthetic set and inspect shape — real work replaces this block with pd.read_csv.

from sklearn.datasets import make_classification
 
X, y = make_classification(
    n_samples=1000, n_features=20, n_informative=10,
    n_redundant=5, n_classes=2, random_state=42
)
df = pd.DataFrame(X, columns=[f"feat_{i}" for i in range(20)])
df['target'] = y
print(df.shape)  # (1000, 21)
df.head(3)

Rendered output: a 5-row preview showing feat_0 ... feat_19 as floats and target as 0/1, with df.shape confirming 21 columns. The generator seed 42 makes every histogram identical across runs.

How do you audit structure, types, and missingness?

Three commands answer "what am I looking at?" before any plot.

print(df.info())        # dtypes + non-null counts
print(df.describe().T)  # count, mean, std, min, quartiles, max
print(df.isna().sum().sort_values(ascending=False).head())

Rendered output: df.info() shows 21 float64 columns, zero nulls in the raw synthetic set — the notebook then injects nulls into two features to demonstrate df.isna().sum() rising to 40-60 rows. describe().T highlights the widest standard deviations, flagging scale differences before modelling.

Next, duplicates and outlier bounds:

print(f"Duplicates: {df.duplicated().sum()}")
# Outlier scan via IQR on one feature
q1, q3 = df['feat_0'].quantile([0.25, 0.75])
iqr = q3 - q1
print(f"IQR bounds: [{q1 - 1.5*iqr:.2f}, {q3 + 1.5*iqr:.2f}]")

How do you handle univariate and bivariate checks?

Sweep each column, then pair it with the target.

# Univariate: histogram + boxplot for one numeric
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
sns.histplot(df['feat_0'], kde=True, ax=axes[0])
sns.boxplot(x=df['feat_0'], ax=axes[1])
plt.show()
 
# Bivariate: feature vs target
sns.boxplot(x='target', y='feat_0', data=df)
plt.show()
 
# Correlation heatmap for top features
corr = df.corr(numeric_only=True).round(2)
sns.heatmap(corr, cmap='coolwarm', annot=False, center=0)
plt.show()

Rendered output: the histogram is right-skewed with a long tail; the boxplot flags ~20 points beyond the whiskers — those are candidates for capping, not blind deletion. The target-conditioned boxplot shows clear separation on feat_0, feat_3, and feat_7.

How do you clean and engineer without leaking?

Cleaning happens after audits, not before.

# Example: median imputation per column (after noting nulls)
for col in ['feat_3','feat_7']:
    df[col] = df[col].fillna(df[col].median())
 
# Clip outliers to IQR bounds instead of dropping
for col in ['feat_0','feat_1']:
    lo, hi = df[col].quantile([0.05, 0.95])
    df[col] = df[col].clip(lower=lo, upper=hi)
 
# Simple feature: interaction flag
df['feat_sum_3'] = df[['feat_0','feat_3','feat_7']].sum(axis=1)

Where to go next with analysis?

Pair this checklist with the Python Pandas guide for DataFrame mechanics, the A/B testing playbook when your EDA ends in an experiment, and the Data Analyst Roadmap for sequencing the full skill stack.

Download the Notebook and Practise

This article is a walkthrough of a runnable Jupyter notebook. Download the original .ipynb and run it locally or on Colab — every code block above appears in order.

Continue your track: Data Analyst Roadmap · Python and Pandas Guide · SQL NULL Handbook · SQL JOIN Fan-Out · Topfolio Practice · Data Analyst vs Engineer

Dataset generators where applicable are in courses/workbooks/generators/ — see citations atop for the exact *.py source for this notebook.

Frequently Asked Questions

What is EDA and why do analysts do it before modelling?

EDA (Exploratory Data Analysis) is a structured inspection of your dataset — shape, types, missing values, outliers, distributions, and relationships — so you catch silent errors like mixed types or leaked targets before any model or dashboard ships.

How do you handle missing values during EDA in Pandas?

Use df.isna().sum() to locate gaps, df.info() to see non-null counts, then decide per column: df.dropna() for disposable rows or df.fillna() / df['col'].fillna(df['col'].median()) when missing means imputable, never blindly filling before a groupby.

What is SettingWithCopyWarning in Pandas?

It fires when you slice a DataFrame and then assign to the slice (df[df['x']>0]['y']=1). Pandas is warning the assignment may hit a copy, not the original. Fix with .loc: df.loc[df['x']>0, 'y']=1.

Which 3 plots catch most data errors?

A histogram for each numeric column (skew/outliers), a countplot for categoricals (rare levels/typos), and a correlation heatmap (multicollinearity or duplicate signals).

Can I reuse this EDA playbook on my own CSV?

Yes — replace the synthetic make_classification block with pd.read_csv('your.csv') and the rest of the checklist (dtypes -> nulls -> dupes -> outliers -> plots) runs unchanged. Download the notebook below.