Redpanda Mini Guide: Kafka API, Checkpoints & Watermarks

# Redpanda Mini Guide: Kafka API, Checkpoints & Watermarks Redpanda is a Kafka-compatible streaming broker (single binary, no ZooKeeper). Spark Structured Streaming reads it with the Kafka connector — same API as Kafka. ## 1. Run Redpanda locally (2 min) ```bash docker run -d --name redpanda -p 9092:9092 \ redpandadata/redpanda:latest redpanda start --smp 1 --overprovisioned rpk topic create shop-clicks -p 3 ``` Produce test events: ```bash echo '{"user":1,"ts":"2026-09-24T10:00:00"}' | rpk topic produce shop-clicks ``` ## 2. Stream into Spark ```python events = (spark.readStream.format("kafka") .option("kafka.bootstrap.servers", "localhost:9092") .option("subscribe", "shop-clicks") .option("startingOffsets", "earliest") .load()) parsed = events.selectExpr("CAST(value AS STRING) AS json") ``` Key options: `startingOffsets` (earliest vs latest), `maxOffsetsPerTrigger` (backpressure), `failOnDataLoss=false` (tolerate retention gaps in dev). ## 3. The two correctness concepts - **Checkpointing** (`option("checkpointLocation", "lake/_chk/clicks")`): exactly-once progress tracking. Delete it and the stream replays from `startingOffsets` — the #1 cause of "duplicate" streams in demos. - **Watermarks** (`.withWatermark("ts", "10 minutes")`): how long to wait for late events in windowed aggregations. Too short drops real lates; too long holds state (memory) forever. ## 4. Windowed count sink ```python from pyspark.sql import functions as F agg = (parsed.withColumn("ts", F.to_timestamp("json.ts")) .withWatermark("ts", "10 minutes") .groupBy(F.window("ts", "5 minutes")) .count()) q = (agg.writeStream.outputMode("append").format("memory") .queryName("click_counts").option("checkpointLocation", "lake/_chk/clicks").start()) ``` ## 5. Mini-lab preview Produce 1,000 click events, stream them through this query into a Delta sink, kill the stream mid-run, restart it, and prove counts are identical (checkpoint replay done right). Full spec in the Redpanda mini-lab lesson.