Class Imbalance — SMOTE, Class Weights, Threshold Adjustment

This lesson on Class Imbalance — SMOTE, Class Weights, Threshold Adjustment is hands-on and example-driven. You will be able to implement and evaluate advanced synthetic data generation techniques, specifically SMOTE and Borderline SMOTE, to effectively mitigate class imbalance in machine learning models. You will learn the mathematical mechanics of interpolation and how to restrict synthetic sample generation to avoid creating noisy data near decision boundaries.

What You'll Be Able To Do

• Implement SMOTE and Borderline SMOTE using Python libraries to balance imbalanced datasets.
• Apply label encoding to categorical features before applying distance-based sampling techniques.
• Articulate the limitations of standard oversampling when minority points are near majority class boundaries.
• Configure K-Nearest Neighbors (KNN) to select appropriate instances for synthetic sample generation.
• Evaluate model performance after applying synthetic oversampling techniques to ensure mitigation of bias.

Detailed Concept Walkthrough

1. SMOTE: Interpolation vs Duplication

SMOTE generates new minority samples by interpolating between existing ones, ensuring the synthetic data maintains the original data distribution's diversity rather than simply duplicating points. This prevents the overfitting that simple oversampling causes.

• Mechanism: SMOTE selects a minority instance (X) and one of its K-Nearest Neighbors (X1) from the same minority class. A new synthetic point (S1) is created along the line segment connecting X and X1.
• Under the Hood: The process relies on distance metrics (like Euclidean distance) to find the KNNs, ensuring that the generated samples are locally consistent with the existing minority cluster structure.
• Interpolation Formula: The new sample S1 is calculated using S1 = X + R · (X - X1), where R is a random number between 0 and 1, controlling the position of S1 along the segment.
• Best Practice: Always apply SMOTE after splitting data into training and testing sets to prevent data leakage, as synthetic samples should only be generated based on the training data.

# Example using imblearn library
from imblearn.over_sampling import SMOTE
from collections import Counter

# Assume X_train and y_train are loaded

sm = SMOTE(random_state=42, k_neighbors=5)
X_res, y_res = sm.fit_resample(X_train, y_train)

# print('Resampled dataset shape %s' % Counter(y_res))

Key Takeaway: SMOTE uses interpolation based on KNN to create diverse synthetic samples, avoiding the overfitting risk of simple duplication.

2. Noise Generation Near Boundaries

Standard SMOTE can fail when minority instances are close to majority instances, leading to synthetic samples that cross the decision boundary or are non-representative noise. This overlap degrades model performance.

• Mechanism: If a minority instance (X) has a majority instance (X2) as a neighbor, standard interpolation might generate a synthetic point (S2) that falls into the majority class region, creating noise.
• Boundary Issue: Simple interpolation assumes local homogeneity, but near the decision boundary, this assumption breaks down, resulting in synthetic points that confuse the classifier.
• Non-Representative Samples: The generated noisy points increase the overlap between classes, making it harder for the model to learn a clean separation boundary, especially for linear classifiers.
• Action: Identify scenarios where interpolation between X and a neighbor X2 (if X2 is majority) results in a non-representative synthetic point (S2).

Key Takeaway: Standard SMOTE generates noise and overlap when minority points are situated near or adjacent to majority class instances.

3. Borderline SMOTE for Cleaner Data

Borderline SMOTE selectively generates synthetic samples only for "borderline examples"—minority points near the decision boundary—and restricts interpolation to only minority-class neighbors. This mitigates the noise problem.

• Mechanism: BSMOTE first identifies minority instances that are difficult to classify (borderline points) by checking how many of their KNNs belong to the majority class.
• Selection Rule: Only these identified borderline points are used as the base (X) for interpolation; non-borderline (safe) points are ignored for generation.
• Restricted Interpolation: When generating a sample from a borderline point X, BSMOTE ensures that the neighbor X1 used for interpolation is also a minority class instance, preventing the synthetic sample from crossing into the majority region.
• Differentiation: Learners must differentiate between safe (interior) minority examples and borderline (boundary) minority examples to apply the restricted interpolation rule effectively.

# Example using imblearn BorderlineSMOTE
from imblearn.over_sampling import BorderlineSMOTE

# BorderlineSMOTE focuses on points near the decision boundary
bsm = BorderlineSMOTE(
    random_state=42, 
    kind='borderline-1', # Common variant
    k_neighbors=5
)
X_res_b, y_res_b = bsm.fit_resample(X_train, y_train)

# BSMOTE is preferred when classes are highly overlapping

Key Takeaway: BSMOTE targets minority points near the boundary and restricts interpolation to minority neighbors, resulting in cleaner synthetic data generation.

4. Preprocessing Categorical Features

Distance-based algorithms like SMOTE require numerical inputs; therefore, categorical variables must be converted using techniques like label encoding before applying the oversampling technique.

• Requirement: SMOTE relies on calculating distances (e.g., Euclidean) between feature vectors to find nearest neighbors and perform interpolation, which is meaningless for non-numeric features.
• Label Encoding: For categorical features with ordinal meaning or when the number of categories is very high, label encoding assigns an integer to each category, preparing the data for distance calculation.
• Workflow Management: The standard workflow involves loading the imbalanced data, dropping irrelevant features (like IDs), performing necessary preprocessing (encoding), and then applying the SMOTE technique to the training set.
• Tooling: Python libraries like Pandas and Scikit-learn provide efficient tools for feature preprocessing within a Jupyter Notebook environment.

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Assume df is the loaded Vehicle Insurance Data
le = LabelEncoder()

# Example: Encoding a categorical column 'Gender'
df['Gender_Encoded'] = le.fit_transform(df['Gender'])

# Drop the original categorical column and irrelevant features
df = df.drop(columns=['Gender', 'ID'])

Key Takeaway: Categorical features must be numerically encoded (e.g., Label Encoding) before applying distance-based synthetic sampling methods like SMOTE.

Topics Covered in Class Imbalance — SMOTE, Class Weights, Threshold Adjustment

1. SMOTE Concept (0:00 - 0:34) — SMOTE generates synthetic data via interpolation to maintain diversity and mitigate overfitting from simple duplication.
2. Interpolation Mechanics (0:35 - 2:12) — Synthetic samples are created along the line segment connecting existing minority examples to visualize a balanced dataset.
3. SMOTE Formula (2:13 - 3:48) — The SMOTE algorithm uses KNN to select neighbors and applies a linear interpolation formula to calculate the new synthetic point.
4. Standard SMOTE Drawbacks (3:50 - 4:46) — Standard SMOTE can generate noisy synthetic examples or introduce class overlap when minority instances are near majority class boundaries.
5. Borderline SMOTE Theory (4:47 - 6:42) — Borderline SMOTE addresses noise by identifying minority points near the decision boundary and restricting interpolation to only minority-class neighbors.
6. Python Setup (6:44 - 9:41) — Practical preparation involves loading imbalanced data, dropping irrelevant features, and preprocessing categorical variables using label encoding.

ML Foundations Cheat Sheet

• SMOTE — Generates synthetic minority samples via interpolation

  sm = SMOTE(random_state=42)
  

• BorderlineSMOTE — Targets boundary points, restricts interpolation to minority neighbors

  bsm = BorderlineSMOTE(kind='borderline-1')
  

• Interpolation Formula — Calculates new sample position between two minority points

  S1 = X + R * (X - X1)
  

• K-Nearest Neighbors (KNN) — Algorithm used by SMOTE to identify local neighbors

  k_neighbors=5
  

• Label Encoding — Converts categorical features to numerical integers for SMOTE

  le.fit_transform(df['Category'])
  

• Data Leakage — Applying SMOTE before train/test split contaminates test set

  sm.fit_resample(X_train, y_train)
  

Comparison Table

Common Pitfalls

• Mistake: Applying SMOTE before splitting data into training and test sets. Avoid: Only fit and resample SMOTE on the training data (X_train, y_train).
• Mistake: Using SMOTE directly on datasets containing non-encoded categorical features. Avoid: Apply Label Encoding or One-Hot Encoding before running the SMOTE algorithm.
• Mistake: Assuming SMOTE is always better than simple duplication. Avoid: Use Borderline SMOTE when classes are highly overlapping to prevent generating noisy samples.

FAQs

• Why does SMOTE use interpolation instead of simple duplication? Interpolation creates diverse, synthetic data points that fill the feature space, which reduces the risk of overfitting compared to duplicating identical samples.
• What is the role of the random number R in the interpolation formula? R is a random number between 0 and 1 that determines how far the new synthetic sample S1 is placed along the line segment between X and X1.
• How does Borderline SMOTE identify "borderline examples"? It uses KNN to check if a minority point has a significant number of majority class neighbors, indicating it is close to the decision boundary.