Deliverable: Git repo with iceberg_job.py, Iceberg table with evolved spec, EVIDENCE.md.
1. Scenario
The analytics team queries by order_date; the fraud team needs order_hour.
Hive partitioning would force a rewrite to change granularity. Use Iceberg.
2. Requirements
- Create Iceberg table
shop.orders(Hadoop or REST catalog, local warehouse dir) partitioned bydays(order_ts). - Load Lab 1 output; record snapshot id + file/channel layout.
- Evolve the spec to
hours(order_ts); append new data; prove old files were NOT rewritten (same file paths in metadata). - Run a hidden-partition query (
WHERE order_ts >= ...) and show partition pruning viaexplain/ scan report. - Demonstrate snapshot rollback:
CALL system.rollback_to_snapshot(or time-travel read) after a bad append. - Document every step with snapshot ids in
EVIDENCE.md.
3. Grading rubric (100 pts)
- Partition evolution proof (30): old files untouched, new writes use hour spec.
- Pruning evidence (25): scan report shows skipped files.
- Snapshot/rollback (20): bad append + successful rollback with counts.
- Code quality (15): catalog config externalized, rerunnable.
- EVIDENCE.md clarity (10).
4. Stretch
Add a schema evolution (new nullable column) and show old snapshots still readable.