Case 4 — Batch to Streaming Migration

# Case 4 — Batch to Streaming Migration ## The brief Your fraud team needs transaction data with under-1-minute lag. Current pipeline is hourly batch. Migrate to streaming without breaking the existing analytics dashboards (which are fine with hourly). ## The setup Current state: - Source: Postgres transactions table. - Ingestion: hourly batch via Airflow (full-table SELECT, MERGE into warehouse). - Warehouse: dbt models build fact_transactions; analytics dashboards read it. - Latency: ~1 hour. Target state: - Two consumers: streaming (fraud team) and batch (analytics, unchanged). - Stream latency: <60 seconds. - Same source of truth: no divergence. ## The task ### Step 1: Pick the architecture (15 min) Options: - **Dual write**: source writes to Postgres AND a Kafka topic. Requires changing the source app (often not allowed). - **CDC to Kafka**: Debezium reads Postgres WAL → Kafka. Streaming consumers read from Kafka. - **CDC to warehouse + materialized view**: CDC to warehouse; warehouse maintains streaming view. Choose CDC to Kafka. Most flexible; most realistic. ### Step 2: Set up Kafka + Debezium (30 min) Use Confluent Cloud free tier or local Docker: - Kafka cluster. - Debezium connector reading Postgres WAL. - Topics: `dbsrv.public.transactions` (one per source table). Verify events flow: insert into Postgres → see message in Kafka topic. ### Step 3: Streaming consumer for fraud (20 min) Python or Java consumer that: - Reads transactions topic. - Applies fraud rules in real time (e.g., transactions over threshold from new account). - Writes to a fraud alerts topic / table. Latency: source insert → fraud alert in under 60s. ### Step 4: Keep analytics pipeline running (15 min) Ensure existing hourly batch pipeline still works: - Option A: keep batch ingestion in parallel. - Option B: replace batch with periodic dump from Kafka topic (preferred — single source of truth). Confirm dbt models still produce same fact_transactions. ### Step 5: Validate (10 min) Verify: - Stream lag < 60s for 99% of events. - Batch numbers match streaming aggregates (no divergence). - Failure of streaming consumer doesn't break batch (decoupled). ## Deliverables - Architecture diagram (before, after). - Code: Debezium config, streaming consumer, batch loader if changed. - Validation report: latency p50/p99, divergence between streaming and batch. - Runbook: "if streaming lag exceeds 5 min, here's what to check". ## What you'll learn - Migration patterns (parallel run, then cutover). - Why CDC to Kafka is the most common production pattern. - Operating Kafka (consumer group lag, replication, retention). - Reconciling two read paths from same source. - Why most teams DON'T migrate to streaming unless forced.