Mastering Pandas Data Cleaning: dropna, fillna & Type Casting

In real-world data analysis, over 70% of your time is spent inspecting, cleaning, and transforming imperfect data. This lesson covers missing value strategies (isna, dropna, fillna), conditional replacement, and type casting with .astype().

---

1. Detecting Missing Values (NaN / None)

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'age': [25, np.nan, 30, 28, np.nan],
    'salary': [70000, 80000, np.nan, 95000, 60000],
    'dept': ['Engineering', 'Marketing', 'HR', np.nan, 'Marketing']
})

# Count missing values per column:
print(df.isna().sum())
# age: 2, salary: 1, dept: 1

# Calculate missing percentage per column:
print((df.isna().mean() * 100).round(1))

---

2. Dropping Missing Data: df.dropna()

Use dropna() when rows or columns with missing values cannot be reliably imputed.

# 1. Drop any row containing at least one NaN:
df_clean = df.dropna(how='any')

# 2. Drop rows only if ALL columns are NaN:
df_clean = df.dropna(how='all')

# 3. Drop rows where specific critical columns are NaN (Best Practice!):
df_clean = df.dropna(subset=['salary', 'name'])

# 4. Drop entire columns that have too many missing values:
df_clean = df.dropna(axis=1, thresh=4)  # Keep columns with at least 4 non-null values

---

3. Imputing Missing Values: df.fillna()

# 1. Fill categorical column with constant string:
df['dept'] = df['dept'].fillna('Unassigned')

# 2. Statistical Imputation (Mean or Median):
# Tip: Use median when data contains salary/revenue outliers!
median_salary = df['salary'].median()
df['salary'] = df['salary'].fillna(median_salary)

# 3. Forward Fill (ffill) & Backward Fill (bfill) for Time Series:
# Copies the previous valid observation forward:
df['age'] = df['age'].ffill()

---

4. Conditional Masking: df.mask() & df.where()

# Replace invalid negative values with NaN:
df['salary'] = df['salary'].mask(df['salary'] < 0, np.nan)

# Replace specific outlier values:
df['age'] = df['age'].mask(df['age'] > 100, 100)

---

5. Type Casting with .astype() & Numeric Coercion

# Converting data types:
df['age'] = df['age'].astype(int)

# Converting string category columns for memory efficiency:
df['dept'] = df['dept'].astype('category')

# Safe numeric conversion when column contains invalid strings (e.g. '$100' or 'N/A'):
prices = pd.Series(['$25.50', '40.00', 'invalid', '15.20'])
clean_prices = pd.to_numeric(prices.str.replace('$', ''), errors='coerce')
# Output: [25.5, 40.0, NaN, 15.2] (invalid converted safely to NaN)