Delta Lab Guide: MERGE, Time Travel & OPTIMIZE

# Delta Lab Guide: MERGE, Time Travel & OPTIMIZE Delta Lake adds an ACID transaction log (`_delta_log/`) on top of Parquet. Every write is an atomic version: readers always see a consistent snapshot. ## 1. Writes that plain Parquet cannot do ```python from delta import configure_spark_with_delta_pip # spark configured with Delta extensions, then: df.write.format("delta").mode("append").save("lake/delta_orders") ``` ## 2. Upserts with MERGE (the money operation) ```sql MERGE INTO delta_orders t USING daily_updates s ON t.order_id = s.order_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *; ``` One statement handles inserts + updates atomically — no duplicate-window hacks. ## 3. Time travel & rollback ```python spark.read.format("delta").option("versionAsOf", 3).load("lake/delta_orders") spark.read.format("delta").option("timestampAsOf", "2026-09-20").load("lake/delta_orders") ``` Bad backfill? `RESTORE TABLE delta_orders TO VERSION AS OF 3` rolls back the log pointer — data files are untouched until VACUUM. ## 4. File health: OPTIMIZE + Z-ORDER + VACUUM - `OPTIMIZE` compacts small files (critical after streaming appends). - `ZORDER BY (customer_id)` co-locates related rows so data-skipping prunes files. - `VACUUM lake/delta_orders RETAIN 168 HOURS` deletes files no longer referenced. Never set retention below your longest time-travel / rollback window. ## 5. Delta vs plain Parquet (interview one-liner) Parquet is a file format; Delta is Parquet + a transaction log giving ACID, MERGE, time travel, and schema enforcement. Same files, plus guarantees.