Lab 3: Iceberg Tables & Partition Evolution

# Lab 3: Apache Iceberg Tables & Partition Evolution (120 min) **Deliverable:** Git repo with `iceberg_job.py`, Iceberg table with evolved spec, `EVIDENCE.md`. ## 1. Scenario The analytics team queries by `order_date`; the fraud team needs `order_hour`. Hive partitioning would force a rewrite to change granularity. Use Iceberg. ## 2. Requirements 1. Create Iceberg table `shop.orders` (Hadoop or REST catalog, local warehouse dir) partitioned by `days(order_ts)`. 2. Load Lab 1 output; record snapshot id + file/channel layout. 3. Evolve the spec to `hours(order_ts)`; append new data; prove old files were NOT rewritten (same file paths in metadata). 4. Run a hidden-partition query (`WHERE order_ts >= ...`) and show partition pruning via `explain` / scan report. 5. Demonstrate snapshot rollback: `CALL system.rollback_to_snapshot` (or time-travel read) after a bad append. 6. Document every step with snapshot ids in `EVIDENCE.md`. ## 3. Grading rubric (100 pts) - Partition evolution proof (30): old files untouched, new writes use hour spec. - Pruning evidence (25): scan report shows skipped files. - Snapshot/rollback (20): bad append + successful rollback with counts. - Code quality (15): catalog config externalized, rerunnable. - EVIDENCE.md clarity (10). ## 4. Stretch Add a schema evolution (new nullable column) and show old snapshots still readable.