Case 6 (Capstone) — Incident Response: Diagnose a Model That Worked Yesterday but Not Today

Enterprise Scenario: At 08:30 on Tuesday morning, executives and support teams escalate a critical production outage: an automated credit card approval model has rejected 88% of applicants over the last 4 hours (normal rejection rate is 18%). The web service reports 100% HTTP 200 uptime, CPU utilization is normal, and latency is healthy at 24ms. However, thousands of qualified applicants are being rejected, costing the company $80,000 per hour. As on-call MLOps engineer, you must execute the Incident Response Playbook: triage telemetry within 15 minutes, execute an immediate atomic rollback to restore business operations, perform forensic data analysis to isolate the silent root cause, and author a postmortem with preventative guardrails.

Technical Requirements & Deliverables

Deliverable 1: Rapid Incident Triage (triage_incident.py)

• Query the production prediction log table without impacting live database operations.
• Compute rolling hourly metrics: rejection percentage, mean predicted credit score, and volume.
• Confirm the severity of the failure and document initial blast radius.

Deliverable 2: Emergency Atomic Rollback Automation (rollback_service.py)

• Execute programmatic rollback via the MLflow Model Registry API:
  • Transition the faulty model version (v4) from Production to Archived.
  • Promote the previous stable version (v3) to Production.
• Provide a secondary fallback: an emergency heuristic rule-based bypass that routes requests safely if the registry is unreachable.

Deliverable 3: Forensic Root-Cause Analysis (isolate_bug.py)

• Analyze raw incoming feature payloads across the pre-incident and post-incident time windows.
• Uncover the silent data pipeline bug: an upstream CRM update overnight altered the column header customer_credit_tenure_months to credit_tenure_months. The downstream feature pipeline received NULL for the expected column, silently coerced missing values to 0 via default imputation, making every applicant appear to have zero credit history.

Deliverable 4: Blameless Postmortem & Preventative Hardening

• Author a formal postmortem report covering:
  • Incident Summary & Executive Impact.
  • Comprehensive Timeline of Events.
  • Root Cause Analysis (The 5 Whys).
  • Where Detection Succeeded and Failed (Why HTTP health checks passed).
• Implement two code guardrails:
  1. Strict Pydantic input contract asserting non-null constraints on critical credit features.
  2. Production Circuit Breaker: A real-time statistical monitor that triggers an automated rollback if rejection rates exceed 35% across any 15-minute rolling window.

Starter Code Scaffold

import mlflow
from mlflow.tracking import MlflowClient
import pandas as pd

client = MlflowClient()

def rollback_model_version(model_name: str = "credit_approval_model"):
    # Atomically rollback Production alias to previous stable version.
    prod_version = client.get_model_version_by_alias(model_name, "Production")
    prev_version = int(prod_version.version) - 1
    
    print(f"Rolling back {model_name}: demoting v{prod_version.version}, promoting v{prev_version}")
    client.set_registered_model_alias(model_name, "Production", str(prev_version))
    client.set_model_version_tag(model_name, str(prod_version.version), "incident_status", "quarantined")

def forensic_null_audit(logs_df: pd.DataFrame, critical_cols: list[str]) -> dict:
    # Analyze null rates and zero-variance anomalies across production feature logs.
    null_rates = logs_df[critical_cols].isnull().mean()
    zero_rates = (logs_df[critical_cols] == 0).mean()
    return {
        "null_rates": null_rates.to_dict(),
        "zero_rates": zero_rates.to_dict()
    }

100-Point Graded Rubric

• Criteria 1: Rapid Incident Triage & Impact Assessment (25 pts)
  • 25 pts: Formulates immediate quantitative assessment from production telemetry; confirms anomaly without modifying live state.
  • 15 pts: Confirms issue but executes invasive live queries or modifies state prematurely.
  • 0 pts: Fails to isolate the timeline or impact of the incident.
• Criteria 2: Atomic Rollback & Circuit Breakers (25 pts)
  • 25 pts: Executes clean programmatic Model Registry rollback in under 2 minutes; implements secondary rule-based fallback.
  • 15 pts: Rollback requires manual container rebuild or lengthy code deploy.
  • 0 pts: Cannot restore service; leaves failing model in production.
• Criteria 3: Forensic Root-Cause Isolation (25 pts)
  • 25 pts: Successfully uncovers silent schema drift (column rename coerced to default zero); proves why traditional software probes failed.
  • 15 pts: Identifies that tenure is zero but does not trace upstream schema rename cause.
  • 0 pts: Attributes failure incorrectly to model overfitting or random drift.
• Criteria 4: Blameless Postmortem & Preventative Hardening (25 pts)
  • 25 pts: Delivers thorough postmortem with 5-Whys; implements strict schema contract and real-time statistical circuit breaker.
  • 15 pts: Author postmortem document but lacks automated code guardrails.
  • 0 pts: Postmortem is superficial or assigns personal blame without systemic defenses.