{ "cells": [ { "cell_type": "markdown", "metadata": { "role": "brief" }, "source": [ "# LAB 2: Visual Storytelling, Statistical Distributions & Heatmaps\n", "**Assignment Version: eda-lab-2.0 \u00b7 6 Tasks \u00b7 23 Rubric Points \u00b7 Estimated Time: 50 min**\n", "\n", "Construct multi-panel Object-Oriented Matplotlib dashboards, statistical Seaborn distributions (hist, box, violin, strip), four-dimension logarithmic campaign bubble charts, and masked Pearson correlation heatmaps.\n", "\n", "### How Marks Are Awarded\n", "| Task | Deliverable | Points |\n", "| :--- | :--- | ---: |\n", "| T13 \u00b7 Build a shared-axis trend dashboard | fig_trend, trend.png, Two labelled shared-axis panels | 5 |\n", "| T14 \u00b7 Show counts, distributions and group shape | fig_distribution, distribution.png, Four correct statistical panels | 4 |\n", "| T15 \u00b7 Distinguish total revenue from Seaborn means | category_summary, fig_category, category_revenue.png, Mean versus total explanation | 3 |\n", "| T16 \u00b7 Encode four campaign dimensions | fig_campaign, campaigns.png, Visual encoding and interpretation | 4 |\n", "| T17 \u00b7 Build and read an executive correlation heatmap | order_features, corr, fig_heatmap, correlations.png | 4 |\n", "| T18 \u00b7 Use pairwise, marginal and faceted views | pair_grid, joint_grid, facet_grid, pairwise.png / joint.png / facets.png | 3 |\n", "\n", "**Total Points Available: 23 points.** (Pass mark: 80%)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Environment Setup\n", "Run unchanged. This initializes your local exercise workspace.\n" ] }, { "cell_type": "code", "metadata": { "role": "setup" }, "source": [ "import sys, json, sqlite3, hashlib\n", "from pathlib import Path\n", "from itertools import combinations\n", "from collections import Counter\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from sqlalchemy import create_engine, text, inspect\n", "from IPython.display import display\n", "ASSIGNMENT_VERSION = \"eda-capstone-2.0\"\n", "SNAPSHOT_DATE = pd.Timestamp(\"2024-07-01\")\n", "ROOT = Path.cwd() / \"topfolio_eda_capstone_v2\"\n", "DATA_DIR, OUTPUT_DIR = ROOT / \"monthly_exports\", ROOT / \"outputs\"\n", "for directory in (DATA_DIR, OUTPUT_DIR):\n", " directory.mkdir(parents=True, exist_ok=True)\n", "sns.set_theme(style=\"whitegrid\", palette=\"colorblind\")\n", "plt.rcParams.update({\"figure.figsize\": (10, 5), \"font.size\": 10})\n", "print(\"Python\", sys.version.split()[0], \"| pandas\", pd.__version__, \"| seaborn\", sns.__version__)\n", "print(\"Workspace:\", ROOT)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Setup & Baselines\n", "Run unchanged. This prepares the exercise environment and data structures.\n" ] }, { "cell_type": "code", "metadata": { "role": "data_setup" }, "source": [ "# Lab 2 Baseline Data Setup: supplies clean sales, orders, monthly trends, and campaigns\n", "rng = np.random.default_rng(42)\n", "products = pd.DataFrame([\n", " (\"P01\", \"Laptop\", \"Electronics\", 899.0), (\"P02\", \"Headphones\", \"Electronics\", 89.0),\n", " (\"P03\", \"USB Cable\", \"Electronics\", 12.0), (\"P04\", \"Desk\", \"Furniture\", 249.0),\n", " (\"P05\", \"Chair\", \"Furniture\", 149.0), (\"P06\", \"Notebook\", \"Office Supplies\", 5.0),\n", " (\"P07\", \"Pen Pack\", \"Office Supplies\", 8.0), (\"P08\", \"T-Shirt\", \"Apparel\", 24.0),\n", " (\"P09\", \"Jacket\", \"Apparel\", 79.0), (\"P10\", \"Kettle\", \"Home & Kitchen\", 39.0),\n", " (\"P11\", \"Mug\", \"Home & Kitchen\", 15.0), (\"P12\", \"Lamp\", \"Furniture\", 49.0),\n", "], columns=[\"product_id\", \"product\", \"product_category\", \"list_price\"])\n", "cities = [(\"Springfield\", \"IL\", \"Midwest\"), (\"Springfield\", \"MA\", \"Northeast\"),\n", " (\"Austin\", \"TX\", \"South\"), (\"Portland\", \"OR\", \"West\")]\n", "customers = pd.DataFrame([(f\"C{i:03}\", *cities[(i-1)%4], [\"Bronze\",\"Silver\",\"Gold\"][(i-1)%3])\n", " for i in range(1,121)], columns=[\"customer_id\",\"city\",\"state\",\"region\",\"customer_tier\"])\n", "\n", "rows = []\n", "product_lookup = products.set_index('product_id')\n", "customer_lookup = customers.set_index('customer_id')\n", "for i in range(720):\n", " cid = f\"C{(i+1 if i < 120 else int(rng.integers(1,121))):03}\"\n", " customer = customer_lookup.loc[cid]\n", " month = i // 120 + 1\n", " timestamp = pd.Timestamp(2024,month,int(rng.integers(1,28)),int(rng.choice([8,10,12,13,17,18,19,20])))\n", " status = \"completed\" if i < 120 or i % 11 else (\"cancelled\" if i % 22 else \"refunded\")\n", " selected = ['P01','P02','P03'] if i%7 == 0 else list(rng.choice(products.product_id,size=int(rng.integers(1,4)),replace=False))\n", " for pid in selected:\n", " product = product_lookup.loc[pid]\n", " quantity = 60 if i == 250 and pid == selected[0] else int(rng.integers(1,5))\n", " rows.append(dict(line_id=f\"L{len(rows)+1:05}\",order_id=f\"O{i+1:04}\",customer_id=cid,\n", " order_date=timestamp,product_id=pid,product=product['product'],\n", " product_category=product.product_category,quantity=quantity,unit_price=float(product.list_price),\n", " status=status,purchase_address=f\"{i+1} Market St, {customer.city}, {customer.state} 12345\",\n", " customer_tier=customer.customer_tier,region=customer.region))\n", "sales = pd.DataFrame(rows)\n", "sales = sales.loc[sales.status.eq('completed')].copy()\n", "sales['line_revenue'] = sales.quantity * sales.unit_price\n", "sales['month'] = sales.order_date.dt.month\n", "sales['hour'] = sales.order_date.dt.hour\n", "sales['city_state'] = sales.purchase_address.str.split(',').str[1].str.strip() + ', ' + sales.purchase_address.str.split(',').str[2].str.strip().str[:2]\n", "orders = sales.groupby('order_id',as_index=False).agg(\n", " customer_id=('customer_id','first'),order_date=('order_date','first'),\n", " city_state=('city_state','first'),region=('region','first'),\n", " customer_tier=('customer_tier','first'),order_revenue=('line_revenue','sum')\n", ")\n", "monthly = orders.groupby(orders.order_date.dt.month).agg(revenue=('order_revenue','sum'), order_count=('order_id','count'))\n", "campaigns = pd.DataFrame(dict(campaign_id=[f'CAM{i:02}' for i in range(1,25)],\n", " marketing_spend=np.geomspace(200,120000,24).round(2),profit_margin=rng.uniform(8,45,24).round(2),\n", " customers_reached=rng.integers(500,20000,24)))\n", "campaigns['revenue'] = (campaigns.marketing_spend*rng.uniform(1.3,8.0,24)).round(2)\n", "print(f\"Environment ready: {len(orders)} orders, {len(sales)} sales lines, {len(campaigns)} campaigns.\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T13", "role": "task_brief" }, "source": [ "## T13 \u00b7 Build a shared-axis trend dashboard\n", "\n", "**5 points \u00b7 about 20 minutes**\n", "\n", "Use the object-oriented Matplotlib API to create `fig_trend` with two vertically stacked Axes and sharex=True: monthly revenue above and distinct order_count below. Use months 1..6, meaningful titles/units, different colours/markers, legends and grids. Save OUTPUT_DIR / 'trend.png' at 300 DPI before show(). All plots must display in the notebook as well as export to disk.\n", "\n", "\n", "**How this task earns marks:**\n", "- 3 points: Correct data and two explicitly targeted shared-axis panels.\n", "- 2 points: Readable labels/units/legend/style and correct high-resolution export.\n" ] }, { "cell_type": "code", "metadata": { "task_id": "T13", "role": "implementation" }, "source": [ "# T13: Build a shared-axis trend dashboard\n", "# Replace the placeholder with your implementation.\n", "raise NotImplementedError('Complete T13')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "task_id": "T13", "role": "public_checks" }, "source": [ "# Public self-checks: keep these; the rubric also checks reasoning and source.\n", "assert len(fig_trend.axes)==2\n", "assert fig_trend.axes[0].get_shared_x_axes().joined(*fig_trend.axes)\n", "assert np.allclose(fig_trend.axes[0].lines[0].get_ydata(),monthly.revenue)\n", "assert np.allclose(fig_trend.axes[1].lines[0].get_ydata(),monthly.order_count)\n", "assert (OUTPUT_DIR/'trend.png').stat().st_size>1000\n", "\n", "print('T13 self-checks passed')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T14", "role": "task_brief" }, "source": [ "## T14 \u00b7 Show counts, distributions and group shape\n", "\n", "**4 points \u00b7 about 20 minutes**\n", "\n", "Create `fig_distribution`, a 2x2 Matplotlib/Seaborn figure at order grain: histogram of order_revenue with KDE; countplot of orders by region; boxplot of order_revenue by region with a light stripplot overlay; violinplot of order_revenue by region with inner quartiles. Label every panel. Keep the valid high-value order visible; use annotation or a clearly labelled scale if needed, not silent deletion. Export distribution.png. Explain the difference between histogram bins, categorical counts, box whiskers and violin density.\n", "\n", "\n", "**How this task earns marks:**\n", "- 2 points: Histogram/KDE and categorical counts use correct populations.\n", "- 2 points: Box/strip/violin plots are correctly labelled and interpret whiskers/density.\n" ] }, { "cell_type": "code", "metadata": { "task_id": "T14", "role": "implementation" }, "source": [ "# T14: Show counts, distributions and group shape\n", "# Replace the placeholder with your implementation.\n", "raise NotImplementedError('Complete T14')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "task_id": "T14", "role": "public_checks" }, "source": [ "# Public self-checks: keep these; the rubric also checks reasoning and source.\n", "assert len(fig_distribution.axes)==4\n", "assert all(ax.get_title() and ax.get_xlabel() and ax.get_ylabel() for ax in fig_distribution.axes)\n", "assert (OUTPUT_DIR/'distribution.png').stat().st_size>1000\n", "\n", "print('T14 self-checks passed')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T14", "role": "written_response" }, "source": [ "### T14 \u00b7 Written response\n", "\n", "Interpret the skew and mean/median, explain boxplot whiskers versus the 1.5\u00d7IQR fences, and explain what violin width adds. State the observation grain for every panel.\n", "\n", "**Your response:**\n", "\n", "_Write your answer here, citing your actual results._\n" ] }, { "cell_type": "markdown", "metadata": { "task_id": "T15", "role": "task_brief" }, "source": [ "## T15 \u00b7 Distinguish total revenue from Seaborn means\n", "\n", "**3 points \u00b7 about 15 minutes**\n", "\n", "Build `category_summary` with product_category,total_revenue,mean_line_revenue,line_count. Create `fig_category` with two horizontal bar panels: total category revenue from the aggregated table, and mean line revenue from raw sales using sns.barplot's default mean estimator. Disable error bars to avoid implying uncertainty for this complete synthetic extract. Label units and the two different statistics. Export category_revenue.png. The financial ranking must use totals, not the mean chart.\n", "\n", "\n", "**How this task earns marks:**\n", "- 2 points: Correct totals and raw-row mean with clear units and ordering.\n", "- 1 points: Explains barplot mean versus preaggregation and countplot frequency.\n" ] }, { "cell_type": "code", "metadata": { "task_id": "T15", "role": "implementation" }, "source": [ "# T15: Distinguish total revenue from Seaborn means\n", "# Replace the placeholder with your implementation.\n", "raise NotImplementedError('Complete T15')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "task_id": "T15", "role": "public_checks" }, "source": [ "# Public self-checks: keep these; the rubric also checks reasoning and source.\n", "assert np.isclose(category_summary.total_revenue.sum(),sales.line_revenue.sum())\n", "assert np.allclose(category_summary.mean_line_revenue*category_summary.line_count,category_summary.total_revenue)\n", "assert len(fig_category.axes)==2 and (OUTPUT_DIR/'category_revenue.png').is_file()\n", "\n", "print('T15 self-checks passed')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T15", "role": "written_response" }, "source": [ "### T15 \u00b7 Written response\n", "\n", "Would sns.barplot(data=sales,x=\"product_category\",y=\"line_revenue\") show category totals? Explain its default statistic and when countplot is the right choice.\n", "\n", "**Your response:**\n", "\n", "_Write your answer here, citing your actual results._\n" ] }, { "cell_type": "markdown", "metadata": { "task_id": "T16", "role": "task_brief" }, "source": [ "## T16 \u00b7 Encode four campaign dimensions\n", "\n", "**4 points \u00b7 about 15 minutes**\n", "\n", "Use the supplied campaigns table to build `fig_campaign`: marketing_spend on x, revenue on y, numeric profit_margin as colour, and customers_reached as bubble area (scale it for readability). Use a logarithmic x-axis because spend spans orders of magnitude, a named colour scale with labelled colourbar, labels with units, and a size explanation/legend. All spends/revenues are positive. Export campaigns.png. Write one evidence-based comparison of two campaigns and one limitation; do not claim attribution/causality from a scatter plot.\n", "\n", "\n", "**How this task earns marks:**\n", "- 2 points: Four correct mappings and log x-axis with positive inputs.\n", "- 1 points: Readable colourbar, size key, units and export.\n", "- 1 points: Accurate comparison and causal limitation.\n" ] }, { "cell_type": "code", "metadata": { "task_id": "T16", "role": "implementation" }, "source": [ "# T16: Encode four campaign dimensions\n", "# Replace the placeholder with your implementation.\n", "raise NotImplementedError('Complete T16')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "task_id": "T16", "role": "public_checks" }, "source": [ "# Public self-checks: keep these; the rubric also checks reasoning and source.\n", "assert fig_campaign.axes[0].get_xscale()=='log'\n", "assert len(fig_campaign.axes)>=2 and len(fig_campaign.axes[0].collections[0].get_offsets())==24\n", "assert (OUTPUT_DIR/'campaigns.png').stat().st_size>1000\n", "\n", "print('T16 self-checks passed')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T16", "role": "written_response" }, "source": [ "### T16 \u00b7 Written response\n", "\n", "Name the mapping of each visual channel. Compare two actual campaign IDs with numbers. What further evidence would be needed before changing marketing spend?\n", "\n", "**Your response:**\n", "\n", "_Write your answer here, citing your actual results._\n" ] }, { "cell_type": "markdown", "metadata": { "task_id": "T17", "role": "task_brief" }, "source": [ "## T17 \u00b7 Build and read an executive correlation heatmap\n", "\n", "**4 points \u00b7 about 20 minutes**\n", "\n", "Build `order_features`, one row per order_id, with order_revenue, line_count (unique lines), units (quantity sum), mean_unit_price, and region. Compute `corr` from the four numeric business metrics only. Create `fig_heatmap`: mask the upper triangle including the diagonal, annotate two decimal places, use a diverging palette centred at zero with vmin=-1/vmax=1, label the colourbar Pearson r, and export correlations.png. Cite two off-diagonal coefficients and explain association versus causation, including any mechanically related metrics.\n", "\n", "\n", "**How this task earns marks:**\n", "- 2 points: Correct order-level metrics and numeric-only correlation matrix.\n", "- 1 points: Correct annotated masked diverging heatmap.\n", "- 1 points: Accurate coefficients and non-causal interpretation.\n" ] }, { "cell_type": "code", "metadata": { "task_id": "T17", "role": "implementation" }, "source": [ "# T17: Build and read an executive correlation heatmap\n", "# Replace the placeholder with your implementation.\n", "raise NotImplementedError('Complete T17')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "task_id": "T17", "role": "public_checks" }, "source": [ "# Public self-checks: keep these; the rubric also checks reasoning and source.\n", "assert order_features.index.is_unique and len(order_features)==len(orders)\n", "assert corr.shape==(4,4) and np.allclose(corr,corr.T) and np.allclose(np.diag(corr),1)\n", "assert np.allclose(corr,order_features[['order_revenue','line_count','units','mean_unit_price']].corr()), 'Use the observed correlations, not a fabricated matrix.'\n", "assert 'order_id' not in corr.columns and 'region' not in corr.columns\n", "assert fig_heatmap.axes[0].collections[0].get_clim()==(-1.0,1.0)\n", "assert (OUTPUT_DIR/'correlations.png').is_file()\n", "\n", "print('T17 self-checks passed')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T17", "role": "written_response" }, "source": [ "### T17 \u00b7 Written response\n", "\n", "Report two actual Pearson coefficients. Explain why an order total and units can correlate by construction and why no coefficient proves a causal effect.\n", "\n", "**Your response:**\n", "\n", "_Write your answer here, citing your actual results._\n" ] }, { "cell_type": "markdown", "metadata": { "task_id": "T18", "role": "task_brief" }, "source": [ "## T18 \u00b7 Use pairwise, marginal and faceted views\n", "\n", "**3 points \u00b7 about 20 minutes**\n", "\n", "Create three separate figures: (1) `pair_grid` from a fixed sample of at most 200 order_features rows, using revenue/units/mean_unit_price, hue=region and histogram diagonals; (2) `joint_grid` of units versus order_revenue with marginal distributions; (3) `facet_grid` of order revenue histograms split by region, sharing x limits so panels are comparable. Use sns.pairplot, sns.jointplot and sns.FacetGrid, respectively. Export pairwise.png, joint.png and facets.png. This is a chart-selection exercise: state which question each view answers, rather than repeating the same conclusion three times.\n", "\n", "\n", "**How this task earns marks:**\n", "- 2 points: Correct distinct pairwise, marginal and regional-facet views with suitable samples/scales.\n", "- 1 points: Explains chart selection and exports three labelled figures.\n" ] }, { "cell_type": "code", "metadata": { "task_id": "T18", "role": "implementation" }, "source": [ "# T18: Use pairwise, marginal and faceted views\n", "# Replace the placeholder with your implementation.\n", "raise NotImplementedError('Complete T18')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "task_id": "T18", "role": "public_checks" }, "source": [ "# Public self-checks: keep these; the rubric also checks reasoning and source.\n", "assert len(pair_grid.data)<=200 and len(facet_grid.axes.flat)==4\n", "assert all((OUTPUT_DIR/name).stat().st_size>1000 for name in ['pairwise.png','joint.png','facets.png'])\n", "\n", "print('T18 self-checks passed')\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "task_id": "T18", "role": "written_response" }, "source": [ "### T18 \u00b7 Written response\n", "\n", "In one sentence per chart explain its analytical purpose and why the facet axis scales should match. Distinguish the overall picture from an outlier-driven pattern.\n", "\n", "**Your response:**\n", "\n", "_Write your answer here, citing your actual results._\n" ] }, { "cell_type": "markdown", "metadata": { "role": "handoff" }, "source": [ "## Submission Checklist\n", "- All 6 tasks attempted; kernel restarted and all cells run cleanly from top to bottom.\n", "- Visual outputs, tables, and written responses retained in the notebook.\n", "- Save and download your completed `.ipynb` file.\n", "- Upload and submit on the corresponding course lab page.\n" ] } ], "metadata": { "language_info": { "name": "python", "version": "3.10" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "topfolio_assignment": { "version": "eda-lab-2.0", "course_slug": "python-eda-data-viz", "title": "LAB 2: Visual Storytelling, Statistical Distributions & Heatmaps", "role": "student_starter", "task_count": 6, "max_points": 23 } }, "nbformat": 4, "nbformat_minor": 4 }