SUBMISSION: Project 3 — Feature Engineering & Preprocessing Pipeline with Scikit-Learn Pipelines

🎯 Hands-on Portfolio Deliverable. Feature engineering decides more leaderboard positions than model choice. This project proves you can build preprocessing that is correct (no leakage), robust (unseen categories don't crash), and measurable (ablate every transformation).

🏢 Business Scenario & Executive Prompt

FreshKart, an online grocer, forecasts next-week demand per SKU-store to cut both stockouts (lost margin) and overstock (spoilage). An intern's notebook scores R² 0.94 — computed with imputation and target-encoding fit on the full dataset. The head of DS asks you: "Rebuild the pipeline so the number is honest, then show me which features actually pay for themselves."

📋 Source Data & Input Schemas

freshkart_demand.csv — 180,000 rows (SKU-store-weeks; mock shape):

Expected shapes: strong promo lift; price_inr missingness correlates with new SKUs; demand is weekly-seasonal.

🎯 Required Deliverables

1. features_pipeline.py — importable build_pipeline() returning an unfitted sklearn Pipeline; no data loading inside.
2. pipeline.pkl — fitted pipeline artifact + feature_list.json (exact post-transform column contract).
3. cv_scores.json — time-based CV scores: baseline (numeric-only) vs +encoding vs +target-encoding vs leaky reference.
4. leakage_report.md — the leaky-vs-clean gap with an explanation of which step leaked and the structural fix.

🛠️ Step-by-Step Implementation Guide

1. Time-based splits. Sort by week; expanding-window folds (train past → validate future). No shuffling — demand has a time axis.
2. Baseline pipeline. Median-impute + scale numerics, one-hot low-cardinality cats, Ridge regression. Record CV MAE/R².
3. Handle sku_id. Never one-hot 2,400 levels: implement in-fold target encoding (smoothed means computed inside each training fold) or hashing. Show the dimensionality math.
4. Prove the leak. Deliberately fit one leaky variant (encoders/imputers on full data) and measure the optimism gap vs the clean pipeline.
5. Ablate. Remove each feature group once; keep only groups that improve time-CV beyond noise. Document the keep/drop table.
6. Harden. handle_unknown="ignore", missing-price path for new SKUs, and a 1-row inference smoke test with an unseen sku_id.

💻 Complete Runnable Starter Scaffold

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import Ridge

NUM = ["price_inr", "promo_flag"]
CAT_LOW = ["store_id", "category"]

def build_pipeline() -> Pipeline:
    pre = ColumnTransformer([
        ("n", Pipeline([("imp", SimpleImputer(strategy="median")),
                        ("sc", StandardScaler())]), NUM),
        ("c", Pipeline([("imp", SimpleImputer(strategy="most_frequent")),
                        ("oh", OneHotEncoder(handle_unknown="ignore"))]), CAT_LOW),
    ])
    return Pipeline([("pre", pre), ("reg", Ridge(alpha=1.0))])

# In-fold target encoding for sku_id (sketch): compute smoothed means with
# statistics from the TRAIN fold only, inside each CV iteration — never globally.
# expanding-window CV: sklearn.model_selection.TimeSeriesSplit

🧪 Acceptance Criteria & Verification Checks

• features_pipeline.py imports cleanly; build_pipeline() returns an unfitted pipeline with no data I/O.
• cv_scores.json shows time-based CV for ≥ 3 variants plus a leaky reference with a documented optimism gap.
• leakage_report.md names the leaking step(s), explains the mechanism, and states the structural fix.
• Unseen sku_id + missing price_inr score without exceptions; feature_list.json matches transform output width.

📊 Evaluation Rubric (100 points — passing ≥ 80)

📤 Submission Guidelines

1. Upload your files (.ipynb, .py, .pkl, .csv, .md, .png) via the file dropzone on this lesson.
2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
3. Name files exactly as listed under Required Deliverables — review checks filenames first.
4. Passing score is 80/100. You may resubmit; the highest score counts.