Statistics for Data Science
Back to Foundations: From Intuition to Mathematics

SUBMISSION: Milestone Project 1 — Exploratory Distribution & Anomaly Analysis on Payment Fraud Data

Profile a 200K-row payments ledger: fit transaction-amount distributions, quantify tail risk with z-scores and IQR fences, and ship an anomaly-flagging notebook with a one-page fraud-ops memo.

135 minutesProject submission
🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Assignment Instructions

🎯 Hands-on Portfolio Deliverable. This is a graded project submission, not a quiz. Your notebook + memo is the artifact you will show in interviews for "tell me about a time you found something in data."

🏢 Business Scenario & Executive Prompt

You are the first data hire at PayRiver, a UPI-adjacent wallet processing ~200K transactions/day. Chargebacks spiked 3× this week. The risk-ops lead believes "a fraud ring is testing stolen cards with micro-charges." The CFO asks: "Is this normal variance or an attack? Which transactions should we freeze right now, and how confident are you?"

You are given one day of ledger data. Deliverables feed directly into the freeze/no-freeze decision due tomorrow 9 AM.

📋 Source Data & Input Schemas

payriver_ledger_day.csv — 200,000 rows (mock shape; generate with the scaffold below):

columndtypedescription
txn_idstrunique transaction id (TXN-000001 …)
tsdatetimetransaction timestamp (IST)
amount_inrfloattransaction amount in ₹ (> 0)
merchant_catcategorygrocery | fuel | electronics | recharge | transfer
channelcategoryapp | web | api
device_age_daysintage of the device fingerprint (0 = brand new)
is_flagged_history0/1prior rule-engine flag (weak signal, precision ≈ 30%)

Expected shapes: heavily right-skewed amount_inr (median ≈ ₹450, mean ≈ ₹1,900, max ≈ ₹4,00,000); device_age_days = 0 for <2% of volume normally.

🎯 Required Deliverables

  1. fraud_eda.ipynb — cleaned, re-runnable top-to-bottom notebook.
  2. flagged_txns.csv — txn_id, amount_inr, z_log, iqr_flag, z_flag, risk_score (0-100), action (freeze \| review \| pass).
  3. fraud_memo.md — one-page ops memo: verdict (attack vs noise), # frozen, expected false-positive cost math.
  4. distributions.png — log-scale histogram + Q-Q subplot figure used in the memo.

🛠️ Step-by-Step Implementation Guide

  1. Load & sanity-check. Parse dtypes, assert 200K rows, no negative amounts, txn_id unique.
  2. Fit the distribution. Plot raw + log-scale histograms of amount_inr. Fit log-normal (scipy lognorm.fit); Q-Q plot vs normal on log(amount). Report median vs mean gap as skew evidence.
  3. Tail math. On log(amount): z-scores; flag |z| > 3. IQR fences on raw amounts. Compare the two flag sets (overlap %).
  4. Behavioral cross-check. Slice flags by device_age_days == 0 and merchant_cat == 'recharge' micro-charges (₹10–₹99): the fraud-ring signature is new devices × micro-charges × burst timestamps.
  5. Risk score. risk = 40·z_flag + 30·iqr_flag + 20·(device_age_days==0) + 10·micro_charge → freeze ≥ 70, review 40–69, else pass.
  6. Cost the decision. Assume freeze costs ₹15/review + 2% false-freeze churn on legit GTV; size the frozen GTV and expected loss in the memo.

💻 Production Starter Scaffold

import numpy as np, pandas as pd
from scipy import stats
import matplotlib.pyplot as plt

# --- 1. Mock ledger (seeded, reproducible) ---
rng = np.random.default_rng(42)
N = 200_000
legit = rng.lognormal(mean=np.log(450), sigma=1.1, size=N)
fraud_ring = rng.uniform(10, 99, size=1_800)  # micro-charge testing
amount = np.concatenate([legit, fraud_ring])
df = pd.DataFrame({
    "txn_id": [f"TXN-{i:06d}" for i in range(len(amount))],
    "amount_inr": np.round(amount, 2),
    "merchant_cat": rng.choice(["grocery","fuel","electronics","recharge","transfer"],
                               size=len(amount), p=[.35,.2,.1,.2,.15]),
    "channel": rng.choice(["app","web","api"], size=len(amount), p=[.6,.25,.15]),
    "device_age_days": rng.choice([0, 5, 40, 300], size=len(amount), p=[.02,.18,.4,.4]),
    "is_flagged_history": rng.choice([0,1], size=len(amount), p=[.97,.03]),
})
df["ts"] = pd.date_range("2026-09-01", periods=len(df), freq="s")

# --- 2. Distribution fit on log(amount) ---
x = np.log(df["amount_inr"])
mu, sigma = x.mean(), x.std()
df["z_log"] = (x - mu) / sigma
q1, q3 = df["amount_inr"].quantile([.25,.75]); iqr = q3-q1
df["iqr_flag"] = df["amount_inr"] > (q3 + 1.5*iqr)
df["z_flag"] = df["z_log"].abs() > 3

# --- 3. Risk score → action ---
df["risk_score"] = (40*df["z_flag"] + 30*df["iqr_flag"]
                    + 20*(df["device_age_days"]==0)
                    + 10*df["amount_inr"].between(10,99)).clip(0,100)
df["action"] = pd.cut(df["risk_score"], [-1,39,69,100], labels=["pass","review","freeze"])
print(df["action"].value_counts())
print("frozen GTV ₹:", round(df.loc[df.action=="freeze","amount_inr"].sum(), 2))

🧪 Acceptance Criteria & Verification Checks

  • Notebook re-runs top-to-bottom with seed 42 and reproduces flagged_txns.csv row counts.
  • distributions.png shows log-scale histogram + Q-Q plot with axis labels and legend.
  • Frozen set is ≥60% micro-charges (₹10–₹99) on zero-age devices (ring signature captured).
  • Memo states freeze count, frozen GTV, and expected false-positive cost with arithmetic shown.
  • flagged_txns.csv has exactly the 7 specified columns; risk_score within 0–100.

📊 Evaluation Rubric (100 points — passing ≥ 80)

#CriterionPointsWhat earns full marks
1Statistical correctness25Right distributions/tests/CIs; assumptions checked, not assumed; math is reproducible
2Code quality & reproducibility25Seeded RNG, clean re-runnable notebook, typed schemas, no hardcoded magic numbers
3Business reasoning & memo25Decision-oriented memo with costed trade-offs; honest about uncertainty and limits
4Visualization & communication25Labeled, honest axes (log-scale where skewed); figures referenced in the memo

📤 Submission Guidelines

  1. Upload your files (.ipynb, .csv, .md, .png) via the file dropzone on this lesson.
  2. Paste your public GitHub repo URL in the submission notes (repo must contain the same files + a 5-line README with the run command).
  3. Name files exactly as listed under Required Deliverables — autograding checks filenames first.
  4. Passing score is 80/100. You may resubmit; the highest score counts.
Checking your access…