Redpanda Mini Guide: Kafka API, Checkpoints & Watermarks

Redpanda is a Kafka-compatible streaming broker (single binary, no ZooKeeper). Spark Structured Streaming reads it with the Kafka connector — same API as Kafka.

1. Run Redpanda locally (2 min)

docker run -d --name redpanda -p 9092:9092 \
  redpandadata/redpanda:latest redpanda start --smp 1 --overprovisioned
rpk topic create shop-clicks -p 3

Produce test events:

echo '{"user":1,"ts":"2026-09-24T10:00:00"}' | rpk topic produce shop-clicks

2. Stream into Spark

events = (spark.readStream.format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "shop-clicks")
  .option("startingOffsets", "earliest")
  .load())
parsed = events.selectExpr("CAST(value AS STRING) AS json")

Note: parsed holds one STRING column (json), so F.to_timestamp("json.ts") would silently yield NULLs — always parse first with from_json + an explicit schema.

Key options: startingOffsets (earliest vs latest), maxOffsetsPerTrigger (backpressure), failOnDataLoss=false (tolerate retention gaps in dev).

3. The two correctness concepts

Checkpointing (option("checkpointLocation", "lake/_chk/clicks")): exactly-once progress tracking. Delete it and the stream replays from startingOffsets — the #1 cause of "duplicate" streams in demos.
Watermarks (.withWatermark("ts", "10 minutes")): how long to wait for late events in windowed aggregations. Too short drops real lates; too long holds state (memory) forever.

4. Windowed count sink

from pyspark.sql import functions as F
schema = "user INT, ts STRING"
typed = (parsed
  .withColumn("doc", F.from_json("json", schema))
  .select("doc.*")
  .withColumn("ts", F.to_timestamp("ts")))
agg = (typed
  .withWatermark("ts", "10 minutes")
  .groupBy(F.window("ts", "5 minutes"))
  .count())
q = (agg.writeStream.outputMode("append").format("memory")
  .queryName("click_counts").option("checkpointLocation", "lake/_chk/clicks").start())

5. Mini-lab preview

Produce 1,000 click events, stream them through this query into a Delta sink, kill the stream mid-run, restart it, and prove counts are identical (checkpoint replay done right). Full spec in the Redpanda mini-lab lesson.