Recap — Why Open Table Formats Matter

# Why Open Table Formats Matter Modern lakehouses are built on a stack of open formats. Knowing the layers — and why they exist — is essential for greenfield 2026 data engineering decisions. ## The stack ``` [ Query Engine: Trino, Snowflake, Spark, Athena ] ↓ [ Table Format: Iceberg, Delta Lake, Hudi ] ← Adds ACID, schema evolution, time travel ↓ [ File Format: Parquet (mostly), ORC, Avro ] ← Columnar, compressed ↓ [ Storage: S3, GCS, Azure Blob, HDFS ] ← Cheap object storage ``` Each layer has a job. ## File format — Parquet A columnar file format with compression. Single file containing rows of data, stored column-by-column with per-column compression. Why Parquet dominates: - **Columnar** — efficient for analytics queries (read only needed columns). - **Compression** — typically 5-10x smaller than CSV. - **Schema embedded** — self-describing. - **Open format** — works with every modern query engine. Other file formats: - **ORC** — similar to Parquet. Hive-era. Less common in modern stacks. - **Avro** — row-oriented. Used for streaming/CDC where row-by-row writes matter. For 99% of analytical workloads: Parquet. ## What's missing from raw Parquet A directory of Parquet files isn't a table: - No ACID transactions (concurrent writes corrupt). - No schema evolution (changing a column breaks readers). - No time travel ("show me data as of yesterday"). - No DELETE/UPDATE (immutable files; you'd rewrite everything). - No partition management. For production, you need a **table format** layered on top. ## Table format — Iceberg / Delta Lake / Hudi Three competing open table formats: ### Apache Iceberg - Originated at Netflix. - Now the de facto open standard (2026). - Supported by Snowflake, BigQuery, Trino, Spark, Athena, Databricks (interop). - Used at Apple, LinkedIn, Adobe. - Maintained by Apache Foundation. ### Delta Lake - Originated at Databricks. - Open-sourced 2019; widely adopted, especially in Databricks ecosystem. - Snowflake added external Delta support in 2024. - Strong choice if you're already on Databricks. ### Apache Hudi - Originated at Uber. - Best-in-class for streaming and frequent updates. - Less common than Iceberg/Delta in new builds. **In 2026**: Iceberg has emerged as the cross-vendor standard. Snowflake, BigQuery, AWS, Cloudflare all support it as external table format. Delta is still strong in Databricks-centric stacks. ## What table formats add ### ACID transactions Multiple writers can update simultaneously. No corrupted reads. Atomic schema changes. ### Schema evolution Add columns, rename columns, change types — without rewriting the table. ```sql ALTER TABLE customers ADD COLUMN segment STRING; -- New column appears; old data has NULL; readers handle gracefully. ``` ### Time travel Query as of any past point. ```sql SELECT * FROM customers FOR SYSTEM_VERSION AS OF '2026-04-01'; -- Or SELECT * FROM customers VERSION AS OF 1234; ``` Useful for audits, debugging, recovery from bad transformations. ### Partition management Logical partitions handled by the table format, not by you. Re-partition without rewriting data. ### Updates and deletes Delete or update rows efficiently (using copy-on-write or merge-on-read strategies). ```sql DELETE FROM customers WHERE created_at < '2024-01-01'; UPDATE customers SET status = 'inactive' WHERE last_login < '2025-01-01'; ``` Without a table format, these operations require rewriting the entire dataset. ## The lakehouse architecture ``` [ Apps generate data ] ↓ [ Iceberg/Delta tables on S3, written in Parquet ] ↓ [ Multiple engines query the same tables ] - Snowflake (external Iceberg tables) - Trino / Athena / Spark - Databricks (Delta or Iceberg via Unity Catalog) ``` The same data, accessible to many engines without copying. One table, many readers. Storage and compute completely decoupled. This is the lakehouse promise: open formats + cheap storage + compute flexibility. ## When lakehouse over pure warehouse Lakehouse advantages: - Less vendor lock-in (data on S3, queryable from multiple engines). - Cheaper storage (S3 vs warehouse-internal storage). - Multiple compute engines for different use cases. - Open formats = future-proof. Lakehouse disadvantages: - More moving parts to operate. - Catalog management (Unity Catalog, Tabular, AWS Glue, project Nessie). - Optimization tuning (compaction, partitioning). - Less mature tooling vs Snowflake/BigQuery for some use cases. For a startup picking a stack today: Iceberg-on-S3 + dbt + Snowflake (querying Iceberg) is increasingly the modern choice. Existing Snowflake-native installations stay native. ## Common lakehouse mistakes 1. **Raw Parquet without a table format.** Production needs ACID; raw Parquet doesn't have it. 2. **Mixing Iceberg, Delta, Hudi without standardization.** Pick one per project. 3. **Underestimating catalog operations.** Iceberg needs a catalog (Glue, Nessie, Tabula