Spark Lakehouse: PySpark, Delta, Iceberg & Streaming
Back to Streaming Mini: Redpanda + Spark

Closing: Lakehouse Checklist & Where Next

Ship-it checklist, interview one-liners, and next steps.

3 minutesInteractive tutorial
🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ship-it checklist (all 4 deliverables)

  • Lab 1: idempotent Parquet ETL, date partitions, README counts + plan.
  • Lab 2: MERGE upserts, time-travel + RESTORE evidence, OPTIMIZE proof.
  • Lab 3: partition evolution without rewrite, pruning + rollback evidence.
  • Redpanda mini: kill/restart with count == 1000, zero duplicates.

The 5 interview one-liners

  1. Lazy transforms + actions; narrow vs wide (shuffle) transforms.
  2. Partition on filter columns with modest cardinality; prune, don't scan.
  3. Parquet = format; Delta/Iceberg = Parquet + log giving ACID/MERGE/travel.
  4. OPTIMIZE compacts, VACUUM deletes — never vacuum inside your travel window.
  5. Iceberg hidden partitioning decouples logical values from paths: evolve freely.

Where next

  • Orchestrate these jobs (Airflow/Dagster) with partition-aware backfills.
  • Serve the gold layer (dbt models on top of Delta/Iceberg).
  • Table health SLIs: file counts, snapshot age, stream lag — alert on drift.

No questions available for this tutorial.

← Back to Redpanda Mini-Lab: Kill, Restart, Prove Exactly-Once