Mastering Pandas Data Cleaning: dropna, fillna, and Type Casting

Mastering Pandas Data Cleaning: dropna, fillna & Type Casting

In real-world data analysis, over 70% of your time is spent inspecting, cleaning, and transforming imperfect data. This lesson covers missing value strategies ( isna , dropna , fillna ), conditional replacement, and type casting with .astype() .

1. Detecting Missing Values (NaN / None)

import pandas as pd import numpy as np

df = pd.DataFrame({ 'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'], 'age': [25, np.nan, 30, 28, np.nan], 'salary': [70000, 80000, np.nan, 95000, 60000], 'dept': ['Engineering', 'Marketing', 'HR', np.nan, 'Marketing'] })

# Count missing values per column: print(df.isna().sum()) # age: 2, salary: 1, dept: 1

# Calculate missing percentage per column: print((df.isna().mean() * 100).round(1))

2. Dropping Missing Data: df.dropna() Use dropna() when rows or columns with missing values cannot be reliably imputed.

# 1. Drop any row containing at least one NaN: df_clean = df.dropna(how='any')

# 2. Drop rows only if ALL columns are NaN: df_clean = df.dropna(how='all')

# 3. Drop rows where specific critical columns are NaN (Best Practice!): df_clean = df.dropna(subset=['salary', 'name'])

# 4. Drop entire columns that have too many missing values: df_clean = df.dropna(axis=1, thresh=4) # Keep columns with at least 4 non-null values

3. Imputing Missing Values: df.fillna()

# 1. Fill categorical column with constant string: df['dept'] = df['dept'].fillna('Unassigned')

# 2. Statistical Imputation (Mean or Median): # Tip: Use median when data contains salary/revenue outliers! median_salary = df['salary'].median() df['salary'] = df['salary'].fillna(median_salary)

# 3. Forward Fill (ffill) & Backward Fill (bfill) for Time Series: # Copies the previous valid observation forward: df['age'] = df['age'].ffill()

4. Conditional Masking: df.mask() & df.where()

# Replace invalid negative values with NaN: df['salary'] = df['salary'].mask(df['salary'] < 0, np.nan)

# Replace specific outlier values: df['age'] = df['age'].mask(df['age'] > 100, 100)

5. Type Casting with .astype() & Numeric Coercion

# Converting data types: df['age'] = df['age'].astype(int)

# Converting string category columns for memory efficiency: df['dept'] = df['dept'].astype('category')

# Safe numeric conversion when column contains invalid strings (e.g. '$100' or 'N/A'): prices = pd.Series(['$25.50', '40.00', 'invalid', '15.20']) clean_prices = pd.to_numeric(prices.str.replace('$', ''), errors='coerce') # Output: [25.5, 40.0, NaN, 15.2] (invalid converted safely to NaN)