Redpanda Mini-Lab: Kill, Restart, Prove Exactly-Once

# Redpanda Mini-Lab: Kill, Restart, Prove Exactly-Once (60 min) **Deliverable:** `stream_job.py` + `produce.py` + `MINI_EVIDENCE.md` with identical pre/post-restart counts. ## 1. Setup Redpanda in Docker (see streaming guide), topic `shop-clicks` with 3 partitions. ## 2. Requirements 1. `produce.py` emits exactly 1,000 JSON click events (user_id, ts, page). 2. `stream_job.py` reads via Kafka connector, parses JSON, dedups on `(user_id, ts)`, writes `append` to a Delta sink with a checkpoint dir. 3. Start stream -> produce 500 -> **kill -9 the stream** -> restart same checkpoint -> produce 500 -> stop gracefully. 4. `MINI_EVIDENCE.md` shows: sink count == 1000, no duplicates on `(user_id, ts)`, and the batch ids proving replay resumed (not restarted). 5. One paragraph: what would change with `startingOffsets=latest` on restart, and why `failOnDataLoss=false` is dev-only. ## 3. Grading rubric (100 pts) - Exactly-once proof (40): count 1000, zero dup keys after kill/restart. - Checkpoint discipline (25): same checkpoint path reused; explanation correct. - Code quality (20): backpressure option set, graceful vs kill path documented. - Write-up (15): latest-vs-earliest paragraph correct. ## 4. Stretch Add a 5-minute tumbling-window aggregation with a 10-minute watermark and show one late event being handled.