Back to Lakehouse Formats: Delta Lake & Iceberg

Apache Iceberg: Spec, Partition Evolution & Catalogs

Snapshots, manifests, hidden partitioning, and multi-engine catalogs.

10 minutesVideo LessonPDF notes

Watch Apache Iceberg: Spec, Partition Evolution & Catalogs on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Partitioning And Pruning — Partitioning lets the query engine skip files that do not match a predicate, which is called pruning.
Old Rigid Directories — Older table formats used physical directories for partitions, making scheme changes expensive and rigid.
Hidden Partitioning — Iceberg moves partition information into metadata and manifest files so data files can live anywhere.
Partition Transforms — Built-in transforms like months, days, bucket, and truncate create partition values without extra columns.
Manifest Pruning Example — A manifest entry stores transformed partition values, letting a March 2025 US query scan exactly one file.
Partition Evolution — You can drop and add partition fields to change the spec for new data without rewriting old files.
Evolution Visual — A query spanning April monthly and May daily partitions scans the whole April partition but prunes May days.
Choosing Partitioning — Choose transforms based on query patterns, avoid over-partitioning, and target 100 MB to 1 GB partitions.

PDF notes

Bottom line: You will be able to explain how Iceberg partitioning enables file pruning, declare hidden partition specs with transforms like months and bucket, and evolve a table's partition layout without rewriting old data. You will also know how to choose partition transforms and target file sizes for real query patterns.

This lesson on Apache Iceberg: Spec, Partition Evolution & Catalogs is hands-on and example-driven. You will be able to explain how Iceberg partitioning enables file pruning, declare hidden partition specs with transforms like months and bucket, and evolve a table's partition layout without rewriting old data. You will also know how to choose partition transforms and target file sizes for real query patterns.

What You'll Be Able To Do

Explain how Iceberg pruning uses manifest metadata to skip irrelevant data files.
Declare hidden partitioning with PARTITIONED BY and built-in transforms such as months, days, bucket, and truncate.
Evolve a partition spec using ALTER TABLE DROP and ADD PARTITION FIELD without rewriting existing files.
Choose time-based, bucket, or truncate transforms based on query predicates and column cardinality.
Target partition sizes between 100 MB and 1 GB to avoid small-file overhead and ineffective pruning.
Query Iceberg tables using raw columns while the engine applies partition transforms automatically.

Detailed Concept Walkthrough

1. Partitioning And Pruning

Partitioning organizes a table into files that can be skipped when a query does not need them. Pruning is the read-path optimization that uses metadata to avoid scanning irrelevant files.

Mechanism: Partitioning maps rows to data files based on column values or transforms. When a query has a predicate, the engine compares it to partition metadata and scans only matching files.
Under the Hood: Iceberg stores partition values in manifest files, not in directory names. The engine reads manifests first, so it can eliminate files before opening Parquet data.
Execution Flow: A query for last Tuesday's sales first evaluates the date predicate, applies the partition transform, matches manifest entries, and then reads only the data files whose partition values overlap the predicate.
Best Practice: Partition on columns that appear in common WHERE clauses. Time-based partitioning is usually the safest starting point for analytic workloads.

-- Without partitioning, the engine scans every file to find last Tuesday
SELECT * FROM sales WHERE sale_date = DATE '2025-03-11';

-- With Iceberg partitioning, metadata enables file pruning
CREATE TABLE sales (
  sale_date DATE,
  region STRING,
  amount DOUBLE
)
USING iceberg
PARTITIONED BY (months(sale_date), region);

Key Takeaway: Partitioning is a performance contract: organize data so the engine can skip files.

2. Hidden Partitioning

Iceberg keeps partition information in the metadata layer, so data files can live anywhere and users query raw columns. This removes the rigid physical directory structure of older table formats.

Mechanism: The table's partition spec declares transforms such as months(sale_date) and region. Iceberg computes the transformed partition values and records them in manifest files.
Under the Hood: There are no required month=03 directories. Data files are tracked by metadata, and the engine uses manifest entries to decide which files to read.
Syntax Rule: Declare partitioning with PARTITIONED BY (months(sale_date), region) in the CREATE TABLE statement. Query with normal predicates on sale_date and region.
Best Practice: Do not add derived columns like sale_month. Iceberg applies the transform automatically, so extra columns are unnecessary and can drift from the spec.

CREATE TABLE prod.db.sales (
  sale_date DATE,
  region STRING,
  amount DOUBLE
)
USING iceberg
PARTITIONED BY (months(sale_date), region);

-- Query uses raw columns; Iceberg applies the transform for pruning
SELECT * FROM prod.db.sales
WHERE sale_date = DATE '2025-03-15' AND region = 'US';

Key Takeaway: Hidden partitioning decouples physical layout from query syntax.

3. Partition Transforms

Partition transforms convert raw column values into partition values. Iceberg provides built-in time transforms, bucket, and truncate to match different data shapes.

Mechanism: months(sale_date) produces a month number such as 627 for March 2025. days(sale_date) produces a day-level value, and years or hours provide coarser or finer time granularity.
Under the Hood: The manifest stores the transformed value, not the raw date. At read time, Iceberg transforms the query predicate the same way and compares it to manifest entries.
Syntax Rule: Use years(col), months(col), days(col), hours(col) for time. Use bucket(N, col) for hashed even distribution. Use truncate(N, col) to group similar string prefixes.
Best Practice: Bucket high-cardinality join keys like customer_id so related rows land in the same files. Truncate high-cardinality product codes to keep partition counts manageable.

-- Time transforms for timestamp columns
PARTITIONED BY (years(event_time), months(event_time), days(event_time), hours(event_time))

-- Bucket high-cardinality customer_id for even distribution
PARTITIONED BY (bucket(16, customer_id))

-- Truncate product_code to first 7 characters to group similar codes
PARTITIONED BY (truncate(7, product_code))

Key Takeaway: Choose the transform that matches your predicate pattern and cardinality.

4. Partition Evolution

Partition evolution lets you change a table's partition spec without rewriting existing data. New data uses the new spec, while old files keep their original partitioning.

Mechanism: ALTER TABLE DROP PARTITION FIELD and ADD PARTITION FIELD update the table metadata. The change applies to future writes, not to existing data files.
Under the Hood: The query engine reads manifests from all partition specs and stitches results together. Pruning efficiency depends on the spec used when each file was written.
Execution Flow: If April is monthly and May is daily, a query for April 14 to May 9 scans the whole April monthly partition but only the first nine daily May partitions.
Best Practice: Evolve partitioning when current partitions become unwieldy. Rewrite old data only if the old layout causes unacceptable query performance.

-- Start with monthly partitions
ALTER TABLE prod.db.sales DROP PARTITION FIELD months(sale_date);
ALTER TABLE prod.db.sales DROP PARTITION FIELD region;
ALTER TABLE prod.db.sales ADD PARTITION FIELD days(sale_date);
ALTER TABLE prod.db.sales ADD PARTITION FIELD region;

-- New data uses days(sale_date); old monthly files remain unchanged

Key Takeaway: Partition evolution changes metadata for new writes, not existing files.

5. Choosing Partitioning And File Size

Partitioning is a performance optimization, so it must follow query patterns. Target partitions between 100 MB and 1 GB to balance pruning and overhead.

Mechanism: Time-based partitioning is the safest bet when data has timestamps because most analytic queries filter by time. Bucket and truncate rescue high-cardinality columns.
Under the Hood: Too many partition columns create many small partitions and small files. Small files increase metadata overhead, open costs, and I/O inefficiency.
Best Practice: Aim for 100 MB to 1 GB per partition. Start with time plus one low-cardinality column, then tune based on real query SLAs.
Nuance: Partitioning on multiple columns can be powerful but is a sharp knife. Monitor file sizes and avoid over-partitioning.

-- Good starting point: monthly time plus region
PARTITIONED BY (months(sale_date), region)

-- Avoid over-partitioning: many columns can create tiny files
-- PARTITIONED BY (days(sale_date), region, bucket(64, customer_id), truncate(7, product_code))

Key Takeaway: Partition for your queries, not for every column.

Topics Covered in Apache Iceberg: Spec, Partition Evolution & Catalogs

Partitioning And Pruning (0:00 - 1:00) — Partitioning lets the query engine skip files that do not match a predicate, which is called pruning.
Old Rigid Directories (1:00 - 2:00) — Older table formats used physical directories for partitions, making scheme changes expensive and rigid.
Hidden Partitioning (2:00 - 3:00) — Iceberg moves partition information into metadata and manifest files so data files can live anywhere.
Partition Transforms (3:00 - 4:00) — Built-in transforms like months, days, bucket, and truncate create partition values without extra columns.
Manifest Pruning Example (4:00 - 5:00) — A manifest entry stores transformed partition values, letting a March 2025 US query scan exactly one file.
Partition Evolution (5:00 - 6:00) — You can drop and add partition fields to change the spec for new data without rewriting old files.
Evolution Visual (6:00 - 7:00) — A query spanning April monthly and May daily partitions scans the whole April partition but prunes May days.
Choosing Partitioning (7:00 - 8:00) — Choose transforms based on query patterns, avoid over-partitioning, and target 100 MB to 1 GB partitions.

Spark Lakehouse Cheat Sheet

PARTITIONED BY (months(sale_date), region) — Declares Iceberg hidden partition spec

CREATE TABLE sales (...) USING iceberg PARTITIONED BY (months(sale_date), region);

months(sale_date) — Time transform for month-level pruning
```
PARTITIONED BY (months(sale_date))
```

days(sale_date) — Time transform for daily partitions

ALTER TABLE sales ADD PARTITION FIELD days(sale_date);

bucket(N, customer_id) — Hash distributes high-cardinality values evenly
```
PARTITIONED BY (bucket(16, customer_id))
```
truncate(N, product_code) — Groups similar string prefixes into partitions
```
PARTITIONED BY (truncate(7, product_code))
```
ALTER TABLE ... DROP PARTITION FIELD — Removes a partition field from spec
```
ALTER TABLE sales DROP PARTITION FIELD months(sale_date);
```
ALTER TABLE ... ADD PARTITION FIELD — Adds new partition field for new data
```
ALTER TABLE sales ADD PARTITION FIELD days(sale_date);
```
Manifest files — Store partition values for pruning

Comparison Table

Aspect	Hive-style partitioning	Iceberg hidden partitioning
Partition location	Physical directories	Metadata in manifest files
Changing scheme	Rewrite or copy data	Update partition spec
Query syntax	Filter partition columns	Filter raw columns
Evolution scope	Whole table rebuild	New data only
Pruning metadata	Directory listing	Manifest partition values

Common Pitfalls

Mistake: Partitioning by too many columns. Avoid: limit to high-value predicates and target 100 MB to 1 GB partitions.
Mistake: Expecting partition evolution to rewrite old files. Avoid: rewrite old data only if its layout hurts query performance.
Mistake: Referencing partition transforms in WHERE clauses. Avoid: filter raw columns and let Iceberg apply transforms automatically.
Mistake: Using daily partitions for low-volume data. Avoid: choose coarser time transforms like months or years.
Mistake: Assuming bucket or truncate changes query syntax. Avoid: still filter the raw column in your WHERE clause.

FAQs

What is hidden partitioning? Iceberg stores partition values in metadata rather than physical directories, so you query raw columns and the engine prunes files automatically.
Does partition evolution rewrite existing data? No. The new spec applies only to new writes; old files keep their original partitioning until you explicitly rewrite them.
How does Iceberg prune files? It reads manifest files that store partition values, compares them to query predicates, and skips data files that cannot contain matching rows.
Which partition transform should I start with? Use time-based transforms like months or days if your queries filter by time; use bucket or truncate for high-cardinality columns.
What file size should partitions target? Aim for 100 MB to 1 GB per partition. Too small adds overhead; too large reduces pruning effectiveness.
Can I query old and new partitions together? Yes. The engine reads manifests from all partition specs and stitches results into one coherent result set.

Common pitfalls

Mistake: Partitioning by too many columns. Avoid: limit to high-value predicates and target 100 MB to 1 GB partitions.
Mistake: Expecting partition evolution to rewrite old files. Avoid: rewrite old data only if its layout hurts query performance.
Mistake: Referencing partition transforms in WHERE clauses. Avoid: filter raw columns and let Iceberg apply transforms automatically.
Mistake: Using daily partitions for low-volume data. Avoid: choose coarser time transforms like months or years.
Mistake: Assuming bucket or truncate changes query syntax. Avoid: still filter the raw column in your WHERE clause.

All right, let's start with the obvious question. In the context of iceberg, what is Partitioning. So imagine a table with billions of rows, very large table, and this is not an uncommon thing in a data lake. And let's say you need all the sales from last Tuesday. Without partitioning, the query engine has to scan every single file to find them. That's not great. That's going to be a lot of scanning. It's going to be very costly, take a long time. With partitioning, the data is organized such that the engine can skip files that don't matter. On the read path, that's called pruning. That's a very important part of how the query engine is going to optimize its work. So think of it as the difference between a clean filing cabinet and a giant pile of papers. Back in the day, partitioning meant creating physical directories for each slice of data, like month equals 0, 1, month equals 0, 2. And so on. It worked, but it was really rigid. You had to build that in to the physical structure of the data, which means if you want to change your partitioning scheme, Good luck with that. You actually have to move things around into new directories. Probably not move, probably actually copy so that the old thing is online while the new thing is coming online. That's a lot. Iceberg takes a different approach, which it calls hidden partitioning. Partition information lives in the metadata layer, specifically in the manifest files. While data files can be physically anywhere. So, no physical directories, no rigid structure, no partitioning scheme that you're committed to when you load the data. That gives you some pretty powerful benefits. Let's once again create a sales table. We're going to partition it by sale date and region. And you can see that there with the partitioned by keyword. Notice something, we're not using sale date directly. We're calling that months function, months on sale date. That's called a partition transform. It's built into Iceberg. It gives you efficient month-level pruning now without having to create a partition for every single date, without having to derive a column that's just got month or any of those shenanigans. You don't need an extra column. Iceberg handles that transform automatically. Here's a manifest file for this table, a little bit simplified for our purposes. There are three entries in the manifest file, each is pointing to a different data file. The first one points to file 001. parquet sitting in the warehouse sales data path in our S3 bucket. Now let's look at the partition values. Instead of storing the raw sale date, Iceberg stores the transformed value, months, sale date, calling that months function, which equals 627. That's March 2025, plus the region, which is U.S., because both those fields were in the partition declaration. When you query the sales table for a specific date, Iceberg knows exactly which partition to check. The date that you want is in the query, that's in your predicates, and the metadata says which dates are going to be Or, more to the point, not going to be in each data file. So, for this query, whose predicates specify March 2025, US region, the engine is going to know to scan exactly one file. And that's going to be that top data file, file0001. parque. There are many built-in partition transformations for time. You saw months. There's also years, days, hours, all really efficient for time-based partitioning. If you have data that you want spread evenly across, say, millions of users or some other very high cardinality kind of column, you can use Bucket. If you have high cardinality product codes, Truncate will group them into manageable chunks. You don't reference any of this in your queries. You're just writing a WHERE clause at the time that you're reading. Iceberg handles the transforms. Automatically and handles the pruning automatically in the read path. Now, what if you have to change a partition? That's partition evolution. And very, very likely you're going to be doing that on A table that's full of data and partitions that are full of data. With older pre-iceberg table formats, that kind of change was a headache. You had to pick a directory structure like year, month, day, or something like that that seemed like good partitioning at the time. When you did that, you were stuck with it until you rebuilt the whole table. Again, physically copying that data around. With Iceberg, it's simple. You can evolve your partitioning any time your needs evolve, potentially without rewrites and hopefully without pain. For example, your sales table is growing like crazy. Good news. Good job. You've done really well. Those monthly partitions are starting to get a little unwieldy, and you might want to switch to daily. With Iceberg, it's a simple partition spec update. You drop both partition fields, as you see here, and then add them back in. In the same order. And the second time around, it's the same sale date column, but of course, a different transform. You've got days instead of months. And the change here, this is key, only applies to new data. Okay, the old monthly partitions, those are data files, and those were partitioned per month. Those are going to stay the way they are unless you do something to rewrite them, which you'd probably rather not do. So scanning on those old monthly partitions, if there's a query with a day predicate. Might not be as efficient as it could be, right? You're still going to have to scan that whole month file. But now for new data going forward with the new Partition scheme, which is days, will have day partitions. We'll have more efficient pruning, more efficient scanning on the new data. Again, at query time, you don't know any of this, it could affect query performance. You're just writing a where clause. At that point, you know nothing about partitioning. You can pretend you know nothing about partitioning. This just applies to the definition of the table itself. Now here's what it looks like visually. Say we've got a query asking for sales between April 14th and May 9th, and we did that repartitioning. At the beginning of May. So April is partitioned by month, and May 1st going forward is partitioned by day. So on the left-hand side, the old monthly partitions, we do have to scan the whole April partition because we want to get to the last two weeks. We've got to get that whole partition because it's all we got. Over on the right, with the daily partitions, we can surgically just get those first nine partitions and ignore ten and after. The blue blocks show us which partitions are pruned and not scanned. The green blocks show us which ones Are scanned. And the engine is, of course, scanning all these different partitions. It will stitch together the results and give you one coherent result set on output. You're probably wondering, okay, this is cool, but how do I decide how to partition my data? Well, it's a good question, and the honest answer, once again, is it depends on. your query patterns. Partitioning is fundamentally a performance optimization, and so you have to know what your queries look like. And what kind of performance SLAs you want for various kinds of predicates in your reads. Time-based partitioning is usually the safest bet if your data has timestamps. Most analytic queries are going to filter by time. If you've ever used a dashboard or done any kind of looking back at business data, you're going to know there's usually a date in there. So that gives you solid pruning and a good place to start. If you have high cardinality columns, there are two transforms that can come to the rescue. Bucket hashes data for even distribution. A bucket is the first one, and what that does is it uses hashes to distribute that high cardinality evenly into buckets. If you're always joining on customer ID and you have a very large number of customers, and like you know, customer ID is always a predicate, well, bucket it and now related rows, that is rows with the same customer ID. Are going to land in the same file. And so that's going to probably lead to better pruning and less scanning. Truncate groups similar values together based on, say, a string. Actual value of a column. If you have a product code that starts with some common string and then has numbers that start to differentiate. You could truncate that to say the first seven characters or something like that. And then those similar codes are going to partition together and end up being in the same data file. Both of those keep your partition counts manageable while enabling the most efficient filtering and pruning possible. You know, you can partition on more than one column at a time. That can be very powerful, but it's a sharp knife. If you go overboard, you might end up with a lot of small. Partitions, and this is just another way to cause the small file problem to rear its ugly head. You don't want that, so you want to be careful with partitioning by too many columns. Again, you just potentially could end up with small files, and that's not a win for anybody. In terms of file size, now is probably a good time to say that you want to aim for partitions that are between 100 megabyte and 1 gigabyte, inside that order of magnitude. Too small, too much below 100 megabyte, you're going to have too much overhead. Too big, pruning doesn't work, lots of I/O, getting those things back and forth. So that's a good sweet spot. Your mileage can always vary. This is a thing that an advanced user might want to really tune. And you always hear cases of people going really big and really small and getting great results. But starting out, if you want a rule of thumb, between 100 megabytes, 1 gigabyte, you'll win. So that is partitioning an iceberg. You're not locked into a single scheme forever. Like with the old formats. You can evolve your partition strategy as your data grows, and that evolution applies to that new data that's ingested over time. Keep old layouts where they are and query everything seamlessly. It's pretty flexible, pretty efficient, leads to some really good performance, and it makes life a whole lot easier as your tables scale.

Frequently asked questions

What is hidden partitioning?

Iceberg stores partition values in metadata rather than physical directories, so you query raw columns and the engine prunes files automatically.

Does partition evolution rewrite existing data?

No. The new spec applies only to new writes; old files keep their original partitioning until you explicitly rewrite them.

How does Iceberg prune files?

It reads manifest files that store partition values, compares them to query predicates, and skips data files that cannot contain matching rows.

Which partition transform should I start with?

Use time-based transforms like months or days if your queries filter by time; use bucket or truncate for high-cardinality columns.

What file size should partitions target?

Aim for 100 MB to 1 GB per partition. Too small adds overhead; too large reduces pruning effectiveness.

Can I query old and new partitions together?

Yes. The engine reads manifests from all partition specs and stitches results into one coherent result set.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.