Module 3 Hands-On Lab: Data Ingestion, Profiling & Systematic Outlier Detection
Execute the complete 6-step Exploratory Data Analysis blueprint on dirty, multi-file retail exports.
Download Starter Notebook: 03_data_cleaning_and_systematic_eda_starter.ipynb
Scenario & Business Context
Northstar Retail has provided six monthly sales CSV exports containing corrupt dates, missing customer IDs, duplicate lines, and repeated header rows. You must profile the dataset, build an auditable cleaning ledger, and perform outlier sensitivity analysis.
What You Will Build & Demonstrate
| Task | Deliverable | Points |
|---|---|---|
| T01 · Combine six monthly exports | File discovery via glob(), batch concatenation, row ledger | 3 |
| T02 · Profile before fixing | .info(), deep memory audit, missingness percentages, duplicate line counts | 3 |
| T03 · Clean with an auditable row ledger | clean_sales, quarantine, stage-by-stage audit counts | 6 |
| T04 · Define valid sales & order grain | Distinguish line items from distinct purchases (sales vs orders) | 3 |
| T09 · Inspect distributions & flag IQR outliers | Tukey IQR calculation ($Q1 - 1.5 \times IQR$, $Q3 + 1.5 \times IQR$) | 4 |
| T10 · Compare keeping, capping and filtering | Outlier sensitivity analysis on total business revenue | 3 |
| T11 · Answer monthly, hourly & geographic questions | Peak purchasing hours, regional sales shares, category leaders | 5 |
| T12 · Count products bought together | Market basket analysis (combinations, Counter) on multi-item orders | 4 |
Total Points Available: 31 points (Pass: 80% / 25 pts)
Submission Guidelines
- Download the starter notebook
.ipynb. - Run the local raw CSV generator cell (generates the 6 monthly files locally).
- Complete the cleaning, profiling, outlier detection, and basket tasks.
- Upload your completed
.ipynbfile below.