Redpanda Mini-Lab: Kill, Restart, Prove Exactly-Once

Deliverable: stream_job.py + produce.py + MINI_EVIDENCE.md with identical pre/post-restart counts.

1. Setup

Redpanda in Docker (see streaming guide), topic shop-clicks with 3 partitions.

2. Requirements

1. produce.py emits exactly 1,000 JSON click events (user_id, ts, page).
2. stream_job.py reads via Kafka connector, parses JSON, dedups on (user_id, ts), writes append to a Delta sink with a checkpoint dir.
3. Start stream -> produce 500 -> kill -9 the stream -> restart same checkpoint -> produce 500 -> stop gracefully.
4. MINI_EVIDENCE.md shows: sink count == 1000, no duplicates on (user_id, ts), and the batch ids proving replay resumed (not restarted).
5. One paragraph: what would change with startingOffsets=latest on restart, and why failOnDataLoss=false is dev-only.

3. Grading rubric (100 pts)

• Exactly-once proof (40): count 1000, zero dup keys after kill/restart.
• Checkpoint discipline (25): same checkpoint path reused; explanation correct.
• Code quality (20): backpressure option set, graceful vs kill path documented.
• Write-up (15): latest-vs-earliest paragraph correct.

4. Stretch

Add a 5-minute tumbling-window aggregation with a 10-minute watermark and show one late event being handled.