{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "role": "brief"
      },
      "source": [
        "# LAB 3: Data Ingestion, Profiling & Systematic Outlier Detection\n",
        "**Assignment Version: eda-lab-3.0 \u00b7 8 Tasks \u00b7 31 Rubric Points \u00b7 Estimated Time: 50 min**\n",
        "\n",
        "Discover and concatenate 6 messy monthly sales CSV exports, profile missingness and duplicates, clean with an auditable row ledger, detect Tukey IQR outliers, run sensitivity comparisons, and calculate market basket co-occurrence.\n",
        "\n",
        "### How Marks Are Awarded\n",
        "| Task | Deliverable | Points |\n",
        "| :--- | :--- | ---: |\n",
        "| T01 \u00b7 Combine six monthly exports | Sorted file list, raw_sales, Per-file row counts | 3 |\n",
        "| T02 \u00b7 Profile before fixing | missing_pct, duplicate_line_count, Structural audit output | 3 |\n",
        "| T03 \u00b7 Clean with an auditable row ledger | clean_sales, quarantine, row_ledger | 6 |\n",
        "| T04 \u00b7 Define valid sales and their grain | sales, orders, Line/order population definition | 3 |\n",
        "| T09 \u00b7 Inspect distributions and flag IQR outliers | order_stats, q1/q3/iqr/bounds, outlier_orders | 4 |\n",
        "| T10 \u00b7 Compare keeping, capping and filtering | sensitivity, Outlier treatment explanation | 3 |\n",
        "| T11 \u00b7 Answer monthly, hourly and geographic questions | monthly, hourly, city_revenue, category_revenue, region_tier_counts | 5 |\n",
        "| T12 \u00b7 Count products bought together | pair_counts, Top five pairs, Duplicate-item self-test | 4 |\n",
        "\n",
        "**Total Points Available: 31 points.** (Pass mark: 80%)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Environment Setup\n",
        "Run unchanged. This initializes your local exercise workspace.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "role": "setup"
      },
      "source": [
        "import sys, json, sqlite3, hashlib\n",
        "from pathlib import Path\n",
        "from itertools import combinations\n",
        "from collections import Counter\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sqlalchemy import create_engine, text, inspect\n",
        "from IPython.display import display\n",
        "ASSIGNMENT_VERSION = \"eda-capstone-2.0\"\n",
        "SNAPSHOT_DATE = pd.Timestamp(\"2024-07-01\")\n",
        "ROOT = Path.cwd() / \"topfolio_eda_capstone_v2\"\n",
        "DATA_DIR, OUTPUT_DIR = ROOT / \"monthly_exports\", ROOT / \"outputs\"\n",
        "for directory in (DATA_DIR, OUTPUT_DIR):\n",
        "    directory.mkdir(parents=True, exist_ok=True)\n",
        "sns.set_theme(style=\"whitegrid\", palette=\"colorblind\")\n",
        "plt.rcParams.update({\"figure.figsize\": (10, 5), \"font.size\": 10})\n",
        "print(\"Python\", sys.version.split()[0], \"| pandas\", pd.__version__, \"| seaborn\", sns.__version__)\n",
        "print(\"Workspace:\", ROOT)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Data Setup & Baselines\n",
        "Run unchanged. This prepares the exercise environment and data structures.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "role": "data_setup"
      },
      "source": [
        "# Run unchanged. Synthetic local data: no credentials or network required.\n",
        "rng = np.random.default_rng(42)\n",
        "products = pd.DataFrame([\n",
        "    (\"P01\", \"Laptop\", \"Electronics\", 899.0), (\"P02\", \"Headphones\", \"Electronics\", 89.0),\n",
        "    (\"P03\", \"USB Cable\", \"Electronics\", 12.0), (\"P04\", \"Desk\", \"Furniture\", 249.0),\n",
        "    (\"P05\", \"Chair\", \"Furniture\", 149.0), (\"P06\", \"Notebook\", \"Office Supplies\", 5.0),\n",
        "    (\"P07\", \"Pen Pack\", \"Office Supplies\", 8.0), (\"P08\", \"T-Shirt\", \"Apparel\", 24.0),\n",
        "    (\"P09\", \"Jacket\", \"Apparel\", 79.0), (\"P10\", \"Kettle\", \"Home & Kitchen\", 39.0),\n",
        "    (\"P11\", \"Mug\", \"Home & Kitchen\", 15.0), (\"P12\", \"Lamp\", \"Furniture\", 49.0),\n",
        "], columns=[\"product_id\", \"product\", \"product_category\", \"list_price\"])\n",
        "cities = [(\"Springfield\", \"IL\", \"Midwest\"), (\"Springfield\", \"MA\", \"Northeast\"),\n",
        "          (\"Austin\", \"TX\", \"South\"), (\"Portland\", \"OR\", \"West\")]\n",
        "customers = pd.DataFrame([(f\"C{i:03}\", *cities[(i-1)%4], [\"Bronze\",\"Silver\",\"Gold\"][(i-1)%3])\n",
        "    for i in range(1,121)], columns=[\"customer_id\",\"city\",\"state\",\"region\",\"customer_tier\"])\n",
        "rows = []\n",
        "product_lookup = products.set_index('product_id')\n",
        "customer_lookup = customers.set_index('customer_id')\n",
        "for i in range(720):\n",
        "    cid = f\"C{(i+1 if i < 120 else int(rng.integers(1,121))):03}\"\n",
        "    customer = customer_lookup.loc[cid]\n",
        "    month = i // 120 + 1\n",
        "    timestamp = pd.Timestamp(2024,month,int(rng.integers(1,28)),int(rng.choice([8,10,12,13,17,18,19,20])))\n",
        "    status = \"completed\" if i < 120 or i % 11 else (\"cancelled\" if i % 22 else \"refunded\")\n",
        "    selected = ['P01','P02','P03'] if i%7 == 0 else list(rng.choice(products.product_id,size=int(rng.integers(1,4)),replace=False))\n",
        "    for pid in selected:\n",
        "        product = product_lookup.loc[pid]\n",
        "        quantity = 60 if i == 250 and pid == selected[0] else int(rng.integers(1,5))\n",
        "        rows.append(dict(line_id=f\"L{len(rows)+1:05}\",order_id=f\"O{i+1:04}\",customer_id=cid,\n",
        "            order_date=timestamp.strftime('%Y-%m-%d %H:%M:%S'),product_id=pid,product=product['product'],\n",
        "            product_category=product.product_category,quantity=str(quantity),unit_price=str(product.list_price),\n",
        "            status=status,purchase_address=f\"{i+1} Market St, {customer.city}, {customer.state} 12345\",\n",
        "            customer_tier=customer.customer_tier,region=customer.region))\n",
        "source = pd.DataFrame(rows)\n",
        "for index,column,value in [(400,'customer_id',None),(401,'customer_id',None),\n",
        "    (450,'order_date','not-a-date'),(451,'order_date','2024-02-30'),\n",
        "    (500,'quantity','unknown'),(501,'quantity','0'),(550,'unit_price','-10'),(551,'unit_price',None),\n",
        "    (600,'product_category',None),(601,'product_category',' electronics ')]:\n",
        "    source.loc[index,column] = value\n",
        "for month in range(1,7):\n",
        "    part = source.loc[pd.to_datetime(source.order_date,errors='coerce').dt.month.eq(month)].copy()\n",
        "    if month == 3:\n",
        "        part = pd.concat([part,source.loc[source.order_date.isin(['not-a-date','2024-02-30'])]],ignore_index=True)\n",
        "    header = pd.DataFrame([dict(zip(source.columns,source.columns))])\n",
        "    blank = pd.DataFrame([{column:None for column in source.columns}])\n",
        "    part = pd.concat([part,part.iloc[[0]],header,blank],ignore_index=True)\n",
        "    part.to_csv(DATA_DIR/f'sales_2024_{month:02}.csv',index=False)\n",
        "campaigns = pd.DataFrame(dict(campaign_id=[f'CAM{i:02}' for i in range(1,25)],\n",
        "    marketing_spend=np.geomspace(200,120000,24).round(2),profit_margin=rng.uniform(8,45,24).round(2),\n",
        "    customers_reached=rng.integers(500,20000,24)))\n",
        "campaigns['revenue'] = (campaigns.marketing_spend*rng.uniform(1.3,8.0,24)).round(2)\n",
        "for name,frame in [('products',products),('customers',customers),('campaigns',campaigns)]:\n",
        "    frame.to_csv(ROOT/f'{name}.csv',index=False)\n",
        "print('Ready: six monthly CSVs; 120 customers; 12 products; 24 campaigns.')\n",
        "del source,rows,part,header,blank\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T01",
        "role": "task_brief"
      },
      "source": [
        "## T01 \u00b7 Combine six monthly exports\n",
        "\n",
        "**3 points \u00b7 about 10 minutes**\n",
        "\n",
        "Discover the six sales_2024_*.csv paths in DATA_DIR, sort them, read each into a list of DataFrames, and concatenate once into `raw_sales` with a fresh index. Keep the untouched extract for audit. Display file names, per-file row counts, total rows and first five records. Do not start by dropping rows.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: All six files read from disk and concatenated once with a fresh index.\n",
        "- 1 points: Source unchanged; file and row counts displayed.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T01",
        "role": "implementation"
      },
      "source": [
        "# T01: Combine six monthly exports\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T01')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T01",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert len(paths) == 6\n",
        "assert len(raw_sales) == sum(len(pd.read_csv(p)) for p in paths), 'Source rows do not reconcile.'\n",
        "assert raw_sales.index.equals(pd.RangeIndex(len(raw_sales)))\n",
        "assert raw_sales.line_id.isna().any() and raw_sales.line_id.eq('line_id').any(), 'Keep raw defects for the audit.'\n",
        "\n",
        "print('T01 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T02",
        "role": "task_brief"
      },
      "source": [
        "## T02 \u00b7 Profile before fixing\n",
        "\n",
        "**3 points \u00b7 about 10 minutes**\n",
        "\n",
        "Show raw_sales.shape, .info(), dtypes, deep memory usage, descending missing-percentage Series `missing_pct`, and `duplicate_line_count` (repeated non-null line IDs after the first occurrence, excluding headers). Print repeated-header and fully-empty row counts separately. Explain why repeated order IDs are not necessarily duplicates.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Shape/types/memory and correct missing percentages.\n",
        "- 1 points: Line-grain duplicates; separate header and empty counts.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T02",
        "role": "implementation"
      },
      "source": [
        "# T02: Profile before fixing\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T02')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T02",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "pd.testing.assert_series_equal(missing_pct.sort_index(),raw_sales.isna().mean().mul(100).sort_index(),check_names=False)\n",
        "assert duplicate_line_count == 6\n",
        "assert missing_pct.is_monotonic_decreasing\n",
        "\n",
        "print('T02 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T02",
        "role": "written_response"
      },
      "source": [
        "### T02 \u00b7 Written response\n",
        "\n",
        "In 2\u20133 sentences distinguish an order, an order line, a duplicate line and a repeated CSV header.\n",
        "\n",
        "**Your response:**\n",
        "\n",
        "_Write your answer here, citing your actual results._\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T03",
        "role": "task_brief"
      },
      "source": [
        "## T03 \u00b7 Clean with an auditable row ledger\n",
        "\n",
        "**6 points \u00b7 about 25 minutes**\n",
        "\n",
        "Create `clean_sales` and `row_ledger` with columns stage, rows_in, rows_removed, rows_out. Apply this order: remove fully empty rows; remove repeated headers (line_id == 'line_id'); deduplicate by line_id keeping first; parse order_date and numeric quantity/unit_price with errors='coerce'; quarantine rows missing customer/date/quantity/price or with quantity/price <= 0. Save rejected rows as `quarantine`. Repair category using product_id in the authoritative products lookup. Keep cancelled/refunded records in clean_sales for audit. Reconcile counts without silent drops.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Empty/header/line-duplicate removal in stated order with reconciliation.\n",
        "- 2 points: Invalid values quarantined; positive typed values and valid dates retained.\n",
        "- 2 points: Categories repaired from product_id; source unchanged.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T03",
        "role": "implementation"
      },
      "source": [
        "# T03: Clean with an auditable row ledger\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T03')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T03",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert list(row_ledger.columns) == ['stage','rows_in','rows_removed','rows_out']\n",
        "assert row_ledger.stage.tolist() == ['empty','headers','duplicates','quarantine']\n",
        "assert (row_ledger.rows_in-row_ledger.rows_removed).eq(row_ledger.rows_out).all()\n",
        "assert row_ledger.rows_in.iloc[0] == len(raw_sales) and row_ledger.rows_out.iloc[-1] == len(clean_sales)\n",
        "assert row_ledger.rows_out.iloc[:-1].tolist() == row_ledger.rows_in.iloc[1:].tolist()\n",
        "assert clean_sales.line_id.is_unique and len(quarantine) == 8\n",
        "assert clean_sales[['customer_id','order_date','quantity','unit_price','product_category']].notna().all().all()\n",
        "assert (clean_sales.quantity > 0).all() and (clean_sales.unit_price > 0).all()\n",
        "assert pd.api.types.is_datetime64_any_dtype(clean_sales.order_date)\n",
        "assert clean_sales.product_category.equals(clean_sales.product_id.map(products.set_index('product_id').product_category))\n",
        "\n",
        "print('T03 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T03",
        "role": "written_response"
      },
      "source": [
        "### T03 \u00b7 Written response\n",
        "\n",
        "Explain each exclusion, why missing category can be repaired, and why a large valid order is not automatically bad data.\n",
        "\n",
        "**Your response:**\n",
        "\n",
        "_Write your answer here, citing your actual results._\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T04",
        "role": "task_brief"
      },
      "source": [
        "## T04 \u00b7 Define valid sales and their grain\n",
        "\n",
        "**3 points \u00b7 about 15 minutes**\n",
        "\n",
        "Create `sales = clean_sales[status == 'completed'].copy()`. Add line_revenue = quantity * unit_price, integer month and hour, and city_state parsed from purchase_address as 'City, ST'. Keep Springfield IL and MA separate. Build `orders`, one row per order_id, with customer_id, order_date, city_state, region, customer_tier and summed order_revenue. Keep sales at line grain. All financial totals and RFM use completed, valid, uncapped sales.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Correct completed population, revenue and temporal/geographic features.\n",
        "- 1 points: Unique order table reconciles without double-counting.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T04",
        "role": "implementation"
      },
      "source": [
        "# T04: Define valid sales and their grain\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T04')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T04",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert sales.status.eq('completed').all() and sales.line_id.is_unique\n",
        "assert np.allclose(sales.line_revenue,sales.quantity*sales.unit_price)\n",
        "assert sales.month.equals(sales.order_date.dt.month) and sales.hour.equals(sales.order_date.dt.hour)\n",
        "assert {'Springfield, IL','Springfield, MA'}.issubset(set(sales.city_state))\n",
        "assert orders.order_id.is_unique and len(orders) == sales.order_id.nunique()\n",
        "assert np.isclose(orders.order_revenue.sum(),sales.line_revenue.sum())\n",
        "assert set(sales.line_id)==set(clean_sales.loc[clean_sales.status.eq('completed'),'line_id']), 'Do not drop valid completed lines.'\n",
        "assert len(orders) < len(sales), 'A multi-line order is still one order.'\n",
        "\n",
        "print('T04 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T09",
        "role": "task_brief"
      },
      "source": [
        "## T09 \u00b7 Inspect distributions and flag IQR outliers\n",
        "\n",
        "**4 points \u00b7 about 15 minutes**\n",
        "\n",
        "At ORDER grain use orders.order_revenue. Display describe(), mean, median and skewness as `order_stats` (Series with mean, median, skew). Compute q1, q3, iqr, lower_bound, upper_bound and `outlier_orders`, using strictly outside Tukey bounds. Show IDs and values. Do not remove outliers from orders or sales; the valid bulk order must remain in official revenue and RFM.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Useful univariate summary and correct order-grain quartiles/bounds.\n",
        "- 2 points: Every and only flagged order included; source population retained.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T09",
        "role": "implementation"
      },
      "source": [
        "# T09: Inspect distributions and flag IQR outliers\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T09')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T09",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert np.isclose(q1,orders.order_revenue.quantile(.25)) and np.isclose(q3,orders.order_revenue.quantile(.75))\n",
        "assert np.isclose(iqr,q3-q1) and np.isclose(lower_bound,q1-1.5*iqr) and np.isclose(upper_bound,q3+1.5*iqr)\n",
        "assert set(outlier_orders.order_id) == set(orders.loc[(orders.order_revenue<lower_bound)|(orders.order_revenue>upper_bound),'order_id'])\n",
        "assert len(outlier_orders)>0 and len(orders)==sales.order_id.nunique()\n",
        "assert outlier_orders.order_id.is_unique, 'Flag each order only once.'\n",
        "assert np.isclose(order_stats['median'],orders.order_revenue.median())\n",
        "\n",
        "print('T09 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T10",
        "role": "task_brief"
      },
      "source": [
        "## T10 \u00b7 Compare keeping, capping and filtering\n",
        "\n",
        "**3 points \u00b7 about 15 minutes**\n",
        "\n",
        "Build `sensitivity` with index keep, cap, filter and columns order_count, revenue, mean_order_value. Keep uses orders unchanged; cap uses clip(lower_bound,upper_bound) on a separate Series; filter excludes flagged orders. Compare the revenue and mean changes. These are sensitivity views: never replace the official sales/orders data or recalculate official RFM from capped values.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Correct keep/cap/filter calculations on separate views.\n",
        "- 1 points: Numerical comparison and reasoned treatment of legitimate bulk orders.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T10",
        "role": "implementation"
      },
      "source": [
        "# T10: Compare keeping, capping and filtering\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T10')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T10",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert set(sensitivity.index)=={'keep','cap','filter'}\n",
        "assert sensitivity.loc['keep','order_count']==len(orders)\n",
        "assert np.isclose(sensitivity.loc['keep','revenue'],sales.line_revenue.sum())\n",
        "assert np.isclose(sensitivity.loc['cap','revenue'],orders.order_revenue.clip(lower_bound,upper_bound).sum())\n",
        "assert sensitivity.loc['filter','order_count']==len(orders)-len(outlier_orders)\n",
        "assert np.isclose(sensitivity.loc['filter','revenue'],orders.loc[~orders.order_id.isin(outlier_orders.order_id),'order_revenue'].sum())\n",
        "\n",
        "print('T10 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T10",
        "role": "written_response"
      },
      "source": [
        "### T10 \u00b7 Written response\n",
        "\n",
        "Compare all three totals using actual numbers. Which population answers financial reporting, which helps describe a typical order, and why is capping not a repair for an invalid customer/date?\n",
        "\n",
        "**Your response:**\n",
        "\n",
        "_Write your answer here, citing your actual results._\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T11",
        "role": "task_brief"
      },
      "source": [
        "## T11 \u00b7 Answer monthly, hourly and geographic questions\n",
        "\n",
        "**5 points \u00b7 about 20 minutes**\n",
        "\n",
        "Create `monthly` with index 1..6 and revenue/order_count (count unique order_id); `hourly` with index 0..23 and order_count, including zero-order hours; `city_revenue` Series by city_state descending; `category_revenue` DataFrame with product_category and total_revenue descending. Use `pd.crosstab` to build `region_tier_counts` at order grain. Identify the peak month, busiest hour and top city using these results, without asserting that more ads will cause more sales.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Monthly/hourly totals and full time indexes at correct grain.\n",
        "- 2 points: City/state and category revenue reconcile, correctly sorted.\n",
        "- 1 points: Order-grain region/tier crosstab.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T11",
        "role": "implementation"
      },
      "source": [
        "# T11: Answer monthly, hourly and geographic questions\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T11')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T11",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert list(monthly.index)==list(range(1,7)) and list(hourly.index)==list(range(24))\n",
        "assert monthly.order_count.sum()==len(orders) and hourly.order_count.sum()==len(orders)\n",
        "assert np.isclose(monthly.revenue.sum(),sales.line_revenue.sum())\n",
        "assert np.isclose(city_revenue.sum(),sales.line_revenue.sum()) and len(city_revenue)==4\n",
        "assert category_revenue.product_category.is_unique and category_revenue.total_revenue.is_monotonic_decreasing\n",
        "for category,revenue in category_revenue.itertuples(index=False,name=None):\n",
        "    assert np.isclose(revenue,sales.loc[sales.product_category.eq(category),'line_revenue'].sum())\n",
        "assert region_tier_counts.to_numpy().sum()==len(orders)\n",
        "\n",
        "print('T11 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "task_id": "T12",
        "role": "task_brief"
      },
      "source": [
        "## T12 \u00b7 Count products bought together\n",
        "\n",
        "**4 points \u00b7 about 15 minutes**\n",
        "\n",
        "Create `pair_counts` as a Counter keyed by alphabetically sorted product-name tuples (A,B), counting how many distinct completed orders contain each pair. Use combinations and Counter. Within each order use unique product names so quantities/repeated lines do not inflate pair frequency; never combine products from different orders. Display the top five pairs. Add a tiny self-test with two baskets: ['Laptop','USB Cable','USB Cable'] and ['Laptop','USB Cable']; the pair must count twice, not three times.\n",
        "\n",
        "\n",
        "**How this task earns marks:**\n",
        "- 2 points: Correct unordered within-order unique-product pairs.\n",
        "- 1 points: Distinct-order counts and deterministic top five.\n",
        "- 1 points: Duplicate-item test passes and counts are not described as causality.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T12",
        "role": "implementation"
      },
      "source": [
        "# T12: Count products bought together\n",
        "# Replace the placeholder with your implementation.\n",
        "raise NotImplementedError('Complete T12')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "task_id": "T12",
        "role": "public_checks"
      },
      "source": [
        "# Public self-checks: keep these; the rubric also checks reasoning and source.\n",
        "assert isinstance(pair_counts,Counter) and len(pair_counts)>0\n",
        "assert pair_self_test[('Laptop','USB Cable')]==2\n",
        "assert all(a<b and 0<count<=len(orders) for (a,b),count in pair_counts.items())\n",
        "assert pair_counts[('Headphones','Laptop')]==sales.groupby('order_id')['product'].agg(lambda p: {'Headphones','Laptop'}.issubset(set(p))).sum()\n",
        "\n",
        "print('T12 self-checks passed')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "role": "handoff"
      },
      "source": [
        "## Submission Checklist\n",
        "- All 8 tasks attempted; kernel restarted and all cells run cleanly from top to bottom.\n",
        "- Visual outputs, tables, and written responses retained in the notebook.\n",
        "- Save and download your completed `.ipynb` file.\n",
        "- Upload and submit on the corresponding course lab page.\n"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python",
      "version": "3.10"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "topfolio_assignment": {
      "version": "eda-lab-3.0",
      "course_slug": "python-eda-data-viz",
      "title": "LAB 3: Data Ingestion, Profiling & Systematic Outlier Detection",
      "role": "student_starter",
      "task_count": 8,
      "max_points": 31
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
