Back to The Modern Data Engineering Landscape

Apache Iceberg & Open Table Formats — ACID on S3

The format wars and why they matter. Iceberg has emerged as the 2026 open lakehouse standard. FIND_VIDEO: search 'apache iceberg delta parquet lakehouse explained' — recommended channel: Tabular / Databricks / Kahan Data Solutions. Aim for 11 min or under.

12 minutesVideo LessonPDF notes

Watch Apache Iceberg & Open Table Formats — ACID on S3 on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Origin of Apache Iceberg — Iceberg was created at Netflix around 2017 to solve data lake reliability challenges and later open-sourced.
ETL vs. ELT Paradigms — Traditional data warehouses enforced strict schemas via ETL, whereas modern lakehouses favor flexible ELT patterns.
Data Lake Inconsistencies — Storing raw files directly in S3 or external tables introduces schema drift and lacks transactional durability.
Data and Manifest Files — Iceberg tracks individual raw data files using intermediate Avro manifest files containing file paths and statistics.
Manifest Lists and Scaling — Manifest lists group multiple manifest files together, reducing the metadata overhead required to manage millions of files.
Metadata File and Snapshots — Metadata JSON files store table snapshots and serve as the single source of truth for historical table versions.
The Catalog Pointer — The catalog maintains the active pointer to the latest metadata file, completing the Iceberg architecture.

PDF notes

Bottom line: You will master the hierarchical architecture of Apache Iceberg and understand how open table formats bring ACID reliability, schema enforcement, and snapshot isolation to object storage like AWS S3. You will be able to trace how queries resolve down through catalogs, metadata files, manifest lists, manifest files, and raw storage files.

This lesson on Apache Iceberg & Open Table Formats — ACID on S3 is hands-on and example-driven. You will master the hierarchical architecture of Apache Iceberg and understand how open table formats bring ACID reliability, schema enforcement, and snapshot isolation to object storage like AWS S3. You will be able to trace how queries resolve down through catalogs, metadata files, manifest lists, manifest files, and raw storage files.

What You'll Be Able To Do

Trace query resolution down through the Iceberg hierarchical metadata tree.
Contrast traditional ETL warehouse patterns with lakehouse open table formats.
Explain snapshot isolation mechanics during write and delete operations.
Identify the structural roles of manifest files, manifest lists, and metadata files.
Evaluate object store reliability trade-offs when shifting from raw data lakes to Iceberg.

Detailed Concept Walkthrough

1. Evolution From Warehouses to Open Table Formats

Data architectures evolved from rigid ETL data warehouses to schema-less object lakes, ultimately requiring open table formats to restore ACID transactional guarantees. Iceberg provides an abstraction layer over cloud object storage without locking data into proprietary engines.

Historical Context: Traditional ETL pipelines used tools like SSIS or Talend to enforce strict schemas and ACID properties inside centralized data warehouses. As data volumes expanded to multi-modal formats, teams shifted toward ELT patterns on cloud object stores like S3.
The Lakehouse Dilemma: Raw data lakes accept diverse file formats such as Parquet, Avro, ORC, and JSON, but they lack native transaction isolation, concurrency control, and schema enforcement. This causes unmanaged external tables to become inconsistent and difficult to maintain.
The Table Format Solution: Open table formats act as a transactional management layer on top of lake storage. They bring database-level reliability, snapshot tracking, and atomic operations directly to object storage files.

-- Example: Creating an Iceberg managed table backed by Parquet on object storage
CREATE TABLE glue_catalog.db.customer_events (
    event_id STRING,
    user_id BIGINT,
    event_type STRING,
    event_ts TIMESTAMP
)
USING iceberg
LOCATION 's3://my-lakehouse-bucket/customer_events';
-- Schema enforcement and snapshot tracking are now enabled automatically

Key Takeaway: Open table formats provide warehouse-grade ACID consistency directly over cheap, scalable cloud object storage.

2. The Iceberg Metadata Hierarchy

Apache Iceberg organizes table state as an immutable tree of metadata spanning catalogs, metadata JSON files, manifest lists, and manifest files. This decoupled design enables atomic commits and snapshot-based time travel.

Catalog Layer: The catalog serves as the single source of truth that atomically points to the latest valid metadata file location. It acts as the entry point for engines resolving table references.
Metadata File Layer: Metadata JSON files maintain table schemas, partition specs, and an array of historical snapshot records (e.g., snapshot 0, snapshot 1). Each table commit produces a new immutable metadata file.
Manifest List Layer: Each snapshot points to a manifest list file (in Avro format), which tracks an array of manifest files along with partition summary statistics for query pruning.
Manifest File Layer: Manifest files track specific raw data files (Parquet, ORC, Avro), storing individual data file paths, partition values, and column-level metric ranges.

/* Iceberg Metadata Hierarchy Resolution:
 * [ Catalog (e.g. AWS Glue, REST, Hive) ]
 *       │ (points to current metadata pointer)
 *       ▼
 * [ Metadata File: v3.metadata.json ] ─── tracks snapshots (s0, s1, s2)
 *       │ (snapshot s2)
 *       ▼
 * [ Manifest List: snap-2-manifest-list.avro ]
 *       ├── Manifest File A (manifest-a.avro) ──► [ data_1.parquet, data_2.parquet ]
 *       └── Manifest File B (manifest-b.avro) ──► [ data_3.orc, data_4.parquet ]
 */

Key Takeaway: Iceberg tracks state through an immutable hierarchical tree of metadata, eliminating expensive object store directory listings.

3. Snapshot Isolation and Atomic State Changes

Every write, update, or delete operation in Iceberg produces a new isolated snapshot rather than mutating existing data in place. Readers always observe a consistent table snapshot while writes occur concurrently.

Mechanism: When new records are appended or deleted, Iceberg writes new raw data files and generates corresponding manifest files. It then writes a new manifest list and a new metadata file containing the latest snapshot ID.
Under the Hood: An atomic catalog swap updates the table reference pointer to the newly written metadata file. If a conflict occurs during the pointer swap, the transaction aborts and retries optimistically.
Time Travel Support: Because prior snapshots and their corresponding metadata trees remain intact on object storage, query engines can read historical table versions without restoring backups.

-- Querying historical snapshots using Iceberg time travel in Spark SQL
SELECT * FROM glue_catalog.db.customer_events
FOR SYSTEM_TIME AS OF '2023-08-01 12:00:00';

-- Alternatively query a specific snapshot ID directly
SELECT * FROM glue_catalog.db.customer_events
FOR SYSTEM_VERSION AS OF 8472910485729104857;

Key Takeaway: Snapshots provide safe concurrent writes and historical time travel by guaranteeing immutability across the metadata tree.

Topics Covered in Apache Iceberg & Open Table Formats — ACID on S3

Origin of Apache Iceberg (0:00 - 0:45) — Iceberg was created at Netflix around 2017 to solve data lake reliability challenges and later open-sourced.
ETL vs. ELT Paradigms (0:45 - 1:55) — Traditional data warehouses enforced strict schemas via ETL, whereas modern lakehouses favor flexible ELT patterns.
Data Lake Inconsistencies (1:55 - 3:05) — Storing raw files directly in S3 or external tables introduces schema drift and lacks transactional durability.
Data and Manifest Files (3:05 - 4:30) — Iceberg tracks individual raw data files using intermediate Avro manifest files containing file paths and statistics.
Manifest Lists and Scaling (4:30 - 5:40) — Manifest lists group multiple manifest files together, reducing the metadata overhead required to manage millions of files.
Metadata File and Snapshots (5:40 - 6:45) — Metadata JSON files store table snapshots and serve as the single source of truth for historical table versions.
The Catalog Pointer (6:45 - 7:50) — The catalog maintains the active pointer to the latest metadata file, completing the Iceberg architecture.

Data Engineering Cheat Sheet

Iceberg Catalog — Stores pointer to current metadata file

spark.sql.catalog.glue_catalog = org.apache.iceberg.spark.SparkCatalog

Metadata File (*.metadata.json) — Defines schemas, partitions, and all snapshots
```
"current-snapshot-id": 1092837465192837465
```
Manifest List (*.avro) — Maps snapshot to its manifest files
```
"manifest_path": "s3://bucket/metadata/manifest-1.avro"
```

Manifest File (*.avro) — Tracks raw data files and metrics

"data_file": {"file_path": "s3://bucket/data/file-1.parquet"}

FOR SYSTEM_VERSION AS OF — Queries table state at snapshot ID

SELECT * FROM db.events FOR SYSTEM_VERSION AS OF 1092837465;

FOR SYSTEM_TIME AS OF — Queries table state at specific timestamp

SELECT * FROM db.events FOR SYSTEM_TIME AS OF '2024-01-01 00:00:00';

Comparison Table

Architecture Layer	Primary Artifact	Core Responsibility
Catalog	Atomic reference pointer	Tracks latest metadata JSON path
Metadata File	JSON schema & snapshot list	Defines table schema and history
Manifest List	Avro list of manifests	Tracks manifests per active snapshot
Manifest File	Avro file index with stats	Maps paths to data files
Data Layer	Parquet, ORC, or Avro files	Stores actual raw record payloads

Common Pitfalls

Mistake: Assuming Iceberg stores data records inside its metadata JSON files. Avoid: Treat metadata files strictly as structural indices that point to external Parquet or ORC files.
Mistake: Relying on directory-level object storage listing to identify table contents. Avoid: Always read table state via Iceberg catalogs and manifest files rather than file system paths.
Mistake: Treating open table formats as an ingestion ETL framework. Avoid: Use Iceberg as a storage abstraction layer managed by compute engines like Spark, Trino, or Flink.

FAQs

Why does Apache Iceberg use multiple layers of metadata instead of one flat file? Hierarchical metadata files and manifest lists allow Iceberg to scale to billions of files without rewriting entire catalog indexes during incremental commits.
What happens to old snapshots when a table is updated? Old snapshots remain accessible in the metadata file until an explicit snapshot expiration or vacuum job cleans up historical files.
Can an Iceberg table combine different physical file formats? Yes, manifest files can reference Parquet, ORC, and Avro files within the same logical table structure.
How does Iceberg achieve ACID atomicity on object storage? Iceberg stages new data and metadata files immutably, then executes a single atomic commit in the catalog pointing to the new metadata JSON.

Common pitfalls

Mistake: Assuming Iceberg stores data records inside its metadata JSON files. Avoid: Treat metadata files strictly as structural indices that point to external Parquet or ORC files.
Mistake: Relying on directory-level object storage listing to identify table contents. Avoid: Always read table state via Iceberg catalogs and manifest files rather than file system paths.
Mistake: Treating open table formats as an ingestion ETL framework. Avoid: Use Iceberg as a storage abstraction layer managed by compute engines like Spark, Trino, or Flink.

So Apache iceberg is open table format that was developed by Netflix around 2017 or thereabout. Now this is now open source and everybody contributes to that. But to better understand how this is important and how useful this particular open table format is especially in the new modern data lake format then we need to understand how this came to be. If you can still recall essentially when we started uh as data engineer with the initial integrations you started with um the good old ELT ETL isn't it ETL so which gives us access to do the extraction from multiple sources transform them with some layers and then load them to some destinations or whatever location you are then come about the ELT which is the extract, load and transform as um the the name connotates. Uh but essentially what this is about is you have all data stored in some data lakehouse format uh data lakehouse and then you can do the transformation and do uh extract what you want essentially from individual files that is inside this um data lakehouse. Then the last bit is the new um open table format which is the F. Uh essentially it's not an integration um paradigm but essentially this is like a a way you interact with the data especially if you're using the ELT uh in these um modern data lake houses. So, ETL essentially you have a central data warehouse which is going to be your your your centralized locations where all the data will be loaded into and you can have as many small data sources in here. It could be a single um it could be a single data warehouse that you're trying to migrate or you're trying to integrate into or some provider are sending you some data or you are doing the extraction yourself wherever it is everything is getting loaded into this particular data warehouse and of course you can have multiple integrations uh between these layers and then eventually load them to some data warehouse. So, of course, this is going to be using something like if you started with something like um SSIS, you can still recall um that works just fine or use something like talent uh or you use something like uh as you go along something like uh Apache Airflow for some of these transformations which is which works just fine and then com DBT uh which helps you to do both ELT and ETL layer. So essentially your data warehouse is getting integrated and loaded into data warehouse period full stop and this essentially enforces some formats like the schema and this helps you to better have better confidence inside of this data like the durability acidity um consistency and stuff like that. Essentially uh data is there. Now uh come the uh we we come to the era of the lake houses where you can have even data warehouses inside your data lakeous and then that gets integrated with multiple data warehouses multiple data files essentially as many data as possible. It could be audio, it could be video, it could be files, it could be JSON, whatever data format can just be loaded into the data lakehouse. This gets messy just pretty fast and your data lakehouse essentially just sits there. You can now do the extraction and do the transformations that you want for visualizations or whatever insights that you want to derive from there. Okay. Now, so we migrated uh essentially into the era of data lakeouses and this is where something like a prominent uh dbt comes into play. you can now uh extract data into or put them inside some external tables. If you're using something like Redshift, put some files into Red Shift um external tables and then do the transformations from there. Uh this works just fine essentially because this is uh more like your S3. I'm just using S3 as like the general versions because you have different versions on uh Azure different version of AWS, Alibaba cloud and the rest of the uh data providers uh cloud providers. All right. So the data sits there you do the uh transformation but this doesn't enforce schema. It doesn't uh because now you have to like be the one to design essentially how you want to present this data. It could be straight from your layout to some visualization layer. It could be from lake lake house to some other um trans uh another another folder inside the same lake house or another table or whatever it is it just gets uh messy really fast you know and it's not as messy is just like the way you want to present the data u essentially now come apache iceberg now apache iceberg as we were saying helps you to understand how best to put the data inside the lakehouse and better connects them uh you know for presentation purposes. Now, this f essentially has uh a barrier in this case for for working with it. So, each of your files that you have in your S3 essentially is going to be somewhere here. So, you can have let's say P, let's say P, you have maybe Havro or you have the OC file. Now it could be as many p as many avo as many RC files maybe support for JSONs and the rest like that but all of these are like the the raw files that you currently have. Now you have a layer on top of this. On top of this layer you are going to have the manifest file and this manifest file is going to be connected to as many of your raw data file that is inside of the lakehouse as possible. So think of let's say 2000s. Yeah. So in this case you could have a single manifest file that contains all the path to all these files that are sitting there whether p aro or oak file. So in this case you're going to have the manifest file. So I'm going to put this I'm going to change the color to separate to you to show that it's um so you have the the manifest file. Now the manifest file could be just connected to just one of this, two of this or a combination of all of them. Then you can also have another manifest file in here which connects to just this one or and this one and another one just connected to the first one. So manifest file manifest file. Okay. So that's how the the structure of uh manifest file looks like. It's it's a little bit messy the way I drew it, but essentially you can have a single manifest file with different combinations of all of these raw data files. Now, on top of this, you have the manifest list. So, I'm going to put this as manifest list. Let me change the color to to blue. So, you have the manifest list. So, I'm going to say this is the manifest list. Manifest list. So, I'll put this as manifest list. Yeah, manifest list. Now, this manifest list can be connected to multiple manifest files. So, you can have let's say your raw files is in hundreds of thousands. Yeah. But your manifest files will be in let's say hundreds or even thousands. Then your manifest list will be in tens or probably hundreds. Then on top of that layer, we continue. But for now just think of as you have the essentially the raw data file uh sitting as in in in their numerosity uh you have them a little bit lesser in the manifest list containing an array of all of these files. So you have the manifest list connected to a single or or just um all of them uh as as you go along. So you can have manifest list one, manifest list two, manifest list three as as many as possible. Now the concept is getting clearer now because you understand the raw file is connected to manifest file. Manifest file is connected to manifest list. Now what creates all of them is called the the metadata file. Now this metadata file is what really sets things um apart. So manifest list and you have the meta meta data file. I'll call it met F. You could call it MFL. You could call it whatever you like. So but the metadata file what it does is connect to a single snapshot of all of these manifest list. So if you have let's say u 10 manifest files inside of a single manifest list. But what that means is if you remove a single file from the directory, you delete it, you add a new one or whatever it is, you've changed the structure. What manifest file is going to do is uh make sure that individual snapshot at a particular time is maintained. So you could have snapshot zero, you could have snapshot zero, snapshot one, snapshot two, as many snapshots as possible inside a single um metadata file. Now this metadata file is like a single source of truth where all the files are connected to the manifest list and the manifest files and essentially the raw files uh as it cascades down uh through throughout the system. Then the final layer in this case will be the catalog. The catalog now this catalog points to the latest manifest metadata file. Yeah. So the latest metadata file is going to be where your catalog is s sitting. Uh think of your table. Yeah, your table is going to be connected to some data in your you know normal uh RDBMS system where you have everything stored in some uh in some tables. Yeah. But inside of paradigm you have the catalog which is the table uh in this case connected to the metadata file. the metadata metadata file that will be the latest metadata file anyways. So that will be connected to the latest snapshots of all of the files in the in the system. So the snapshot will now be connected to the metadata list uh manifest list and the manifest list connected to the manifest file manifest file to the to the raw data. So you have this connected to or pointing pointing directly to the metadata file in this case which is snapshot zero snapshot one and this snapshot zero can just be part of this. This could be just this one and um this could be this and this could be that. But inside of that you you you have essential control of what happens in the past what is currently happening or and what has maybe happened so much uh far back in the past. So the the present situation of the the the the data lake is maintained inside of this metadata file and you go down to the meta manifest list then the manifest file then the the raw data file. So this is going to be your table uh uh which is pointing to the latest manifest file. So essentially this is um what the architecture of um Apache iceberg look like uh under the hood. So, I know this is kind of interesting, especially if this is your first time of getting to know what Apache iceberg means.

Frequently asked questions

Why does Apache Iceberg use multiple layers of metadata instead of one flat file?

Hierarchical metadata files and manifest lists allow Iceberg to scale to billions of files without rewriting entire catalog indexes during incremental commits.

What happens to old snapshots when a table is updated?

Old snapshots remain accessible in the metadata file until an explicit snapshot expiration or vacuum job cleans up historical files.

Can an Iceberg table combine different physical file formats?

Yes, manifest files can reference Parquet, ORC, and Avro files within the same logical table structure.

How does Iceberg achieve ACID atomicity on object storage?

Iceberg stages new data and metadata files immutably, then executes a single atomic commit in the catalog pointing to the new metadata JSON.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.