Pipeline + SHAP Walkthrough

This lesson on Pipeline + SHAP Walkthrough is hands-on and example-driven. You will construct an end-to-end scikit-learn preprocessing and gradient boosting pipeline for SaaS churn prediction and explain its predictions using TreeSHAP. You will extract post-transformation feature names to map SHAP values accurately back to raw business attributes and diagnose key drivers of customer churn.

What You'll Be Able To Do

• Construct an end-to-end ColumnTransformer and Pipeline integrating categorical encoders and gradient boosted trees.
• Extract transformed feature names from the fitted preprocessor to align SHAP attribution matrices.
• Compute TreeSHAP explanation values using shap.TreeExplainer on the fitted pipeline estimator.
• Generate summary and waterfall plots to communicate global and local churn risk drivers to stakeholders.

Detailed Concept Walkthrough

1. Leak-Free Preprocessing Pipeline Architecture

Encapsulate feature transformations and tree-based estimators into a unified scikit-learn Pipeline to guarantee strict separation between training and evaluation splits.

• Mechanism: A ColumnTransformer applies targeted transformations—such as OneHotEncoder for nominal variables and StandardScaler or passthrough operations for numerical metrics—independently across feature subsets.
• Execution Flow: Nesting the transformer within a Pipeline ensures that all stateful parameters (like imputation medians and category vocabularies) are learned exclusively during .fit() on the training folds.
• Best Practice: Set remainder='passthrough' or explicitly define transformations for every column in the schema to avoid unintentional feature drops.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from lightgbm import LGBMClassifier

num_cols = ['tenure_months', 'monthly_charges', 'support_tickets']
cat_cols = ['contract_type', 'payment_method']

preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), num_cols),
        ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), cat_cols)
    ]
)

churn_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', LGBMClassifier(n_estimators=100, random_state=42))
])

Key Takeaway: Pipelines prevent data leakage by isolating transformation fitting to the training partition.

2. Feature Name Alignment for Model Explanations

TreeSHAP requires direct access to the underlying tree model and the transformed feature matrix, necessitating clean recovery of expanded column headers.

• Mechanism: Calling get_feature_names_out() on the fitted ColumnTransformer retrieves the transformed column names, including expanded one-hot categorical labels.
• Under the Hood: Tree estimators operate purely on numeric arrays, stripping column metadata during inference unless explicitly reconstructed for SHAP interpretability.
• Best Practice: Transform validation feature sets using pipeline.named_steps['preprocessor'].transform(X_val) before passing them into the SHAP explainer.

# Fit the pipeline and transform validation data
churn_pipeline.fit(X_train, y_train)
X_val_trans = churn_pipeline.named_steps['preprocessor'].transform(X_val)

# Retrieve one-hot encoded and scaled feature names
feature_names = churn_pipeline.named_steps['preprocessor'].get_feature_names_out()

# Wrap in DataFrame for named SHAP rendering
import pandas as pd
X_val_df = pd.DataFrame(X_val_trans, columns=feature_names, index=X_val.index)

Key Takeaway: Always extract feature names directly from the fitted transformer to prevent misattributed SHAP values.

3. TreeSHAP Attribution and Interpretability Analysis

TreeSHAP computes exact Shapley values in polynomial time to explain individual churn probabilities and cohort-level feature importance.

• Mechanism: shap.TreeExplainer traverses the ensemble structure of the fitted tree model to calculate additive feature contributions against a baseline expected value.
• Under the Hood: The sum of all SHAP values plus the base expected value equals the raw margin model prediction for any given observation.
• Execution Flow: Global summary plots (shap.summary_plot) rank overall churn drivers, while local waterfall plots (shap.plots.waterfall) explain individual customer churn triggers.

import shap

# Initialize explainer on the fitted classifier
model = churn_pipeline.named_steps['classifier']
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_val_df)

# Global feature impact summary
shap.summary_plot(shap_values.values, X_val_df, show=False)

# Local explanation for the highest-risk churn candidate
high_risk_idx = 0
shap.plots.waterfall(shap_values[high_risk_idx])

Key Takeaway: TreeSHAP provides mathematically consistent local and global explanations directly from tree ensembles.

Topics Covered in Pipeline + SHAP Walkthrough

1. Pipeline Architecture Setup (0:00 - 2:15) — Constructing the ColumnTransformer and Pipeline to bundle feature transformations with the gradient boosted classifier.
2. Model Training & Feature Extraction (2:15 - 5:00) — Fitting the churn model and extracting transformed column names using get_feature_names_out.
3. TreeSHAP Global Explanations (5:00 - 8:10) — Initializing TreeExplainer and generating beeswarm summary plots to diagnose population-level churn drivers.
4. Local Churn Diagnostics (8:10 - 11:00) — Plotting individual customer waterfall charts to explain why specific accounts are flagged as high churn risk.

ML Case Labs Cheat Sheet

• ColumnTransformer(transformers=[...]) — Applies distinct transformations to specified column subsets

  ct = ColumnTransformer([('num', StandardScaler(), ['tenure'])])
  

• Pipeline(steps=[...]) — Chains sequential data transformers and a final estimator

  pipe = Pipeline([('prep', ct), ('clf', LGBMClassifier())])
  

• transformer.get_feature_names_out() — Extracts transformed column names including encoded categories

  cols = pipe.named_steps['prep'].get_feature_names_out()
  

• shap.TreeExplainer(model) — Initializes fast exact Shapley explainer for tree models

  explainer = shap.TreeExplainer(pipe.named_steps['clf'])
  

• shap.summary_plot(shap_values, X) — Renders global feature impact and directionality distribution

  shap.summary_plot(shap_vals.values, X_val_trans)
  

• shap.plots.waterfall(shap_values[i]) — Visualizes step-by-step feature attributions for a single instance

  shap.plots.waterfall(shap_values[0])
  

Comparison Table

Common Pitfalls

• Mistake: Passing raw untransformed features into TreeExplainer when using a Pipeline. Avoid: Extract the preprocessed array using the fitted preprocessor transformer before computing SHAP values.
• Mistake: Fitting the preprocessor separately on the full dataset before splitting. Avoid: Encapsulate transformations inside a Pipeline or fit exclusively on the training split.
• Mistake: Using TreeExplainer on non-tree steps in a Pipeline. Avoid: Target the fitted tree estimator explicitly via named_steps classifier.

FAQs

• Why does TreeExplainer require transformed features rather than raw data frames? Tree models execute splits on encoded numeric values, so SHAP matrices must match the internal dimensionality and column order of the fitted estimator.
• How do you interpret negative SHAP values in a binary churn model? Negative SHAP values push the prediction below the baseline expected value, indicating that the feature value reduces the customer's likelihood of churning.
• What happens if a categorical column contains unseen levels during evaluation? Setting handle_unknown='ignore' in OneHotEncoder ensures unseen categories produce zero vectors rather than raising runtime errors.