Case 6 — Lakehouse Migration (Move Data to Iceberg)

# Case 6 — Lakehouse Migration ## The brief A large fact table is expensive to maintain in the warehouse. Migrate it to an Iceberg-format lakehouse table on S3, where multiple engines (Spark, Trino, Athena, even the warehouse) can read it. ## The setup Source: `warehouse.fact_events` — 500GB, 5B rows, partitioned by day. Pain points: - Storage costs $X/month in Snowflake. - Queries on historical data are slow + expensive. - ML team wants to read it with Spark; currently does double-load. Target: Iceberg table on S3 with Glue/Polaris catalog. ## The task ### Step 1: Set up Iceberg + catalog (20 min) Choose: - **AWS Glue** as catalog (managed, easy). - **Apache Polaris** (open source, more flexible). - **Lakekeeper** (newer; growing). Configure: - S3 bucket for table data. - Glue database `lakehouse`. - Spark or Trino client configured with the catalog. ### Step 2: Migrate data (30 min) Two approaches: Option A: Spark COPY from warehouse: ```python df = spark.read.format("snowflake").option("table", "fact_events").load() df.writeTo("lakehouse.events").partitionedBy("event_date").create() ``` Option B: Snowflake EXPORT to S3 + Iceberg register: ```sql COPY INTO 's3://lake/fact_events/' FROM fact_events FILE_FORMAT = (TYPE = parquet); -- Then register in Iceberg via Spark or Trino CREATE EXTERNAL TABLE events ...; ``` Choose Option A for simplicity. Run; expect ~1 hour for 500GB. ### Step 3: Set up incremental sync (20 min) Daily job that: - Reads new partitions from Snowflake. - Appends to Iceberg table. ```python new_data = spark.sql(''' SELECT * FROM fact_events WHERE event_date = current_date - INTERVAL 1 DAY ''') new_data.writeTo("lakehouse.events").append() ``` Idempotent: re-running for same date should overwrite, not duplicate. Use `replaceWhere`: ```python new_data.writeTo("lakehouse.events").overwritePartitions() ``` ### Step 4: Migrate downstream readers (15 min) For each downstream consumer (dashboards, ML pipelines, dbt models): - Update SQL to read from Iceberg table. - Run side-by-side for 1 week; reconcile any differences. ### Step 5: Decommission (10 min) After validation, drop the warehouse table. Update budget projections. ## Deliverables - Architecture: before (warehouse only) vs after (lakehouse + warehouse). - Migration script. - Daily incremental script. - Cost analysis: before vs after. - Reconciliation report: warehouse table vs Iceberg table, any differences and why. ## What you'll learn - Real Iceberg / Spark patterns. - Catalog choice (Glue, Polaris, Lakekeeper). - Why lakehouse separates storage and compute. - Incremental sync gotchas (partition overwrite, time travel). - Cost analysis is essential to justify the migration. - Decommissioning legacy systems is the hardest part.