Back to Lakehouse Formats: Delta Lake & Iceberg

Delta Lake: ACID, Time Travel & Merges

Transaction log, MERGE upserts, time travel, and OPTIMIZE/Z-order.

9 minutesVideo LessonPDF notes

Watch Delta Lake: ACID, Time Travel & Merges on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Intro and Promise — Naveen introduces the lesson and promises a technical, under-the-hood look at how Delta Lake implements ACID.
The Four ACID Properties — Atomicity, Consistency, Isolation, and Durability are each explained with a bank-transfer style analogy.
Why Data Lakes Need ACID — A horror story of three concurrent ETL jobs corrupting a lake shows what breaks without transactions.
Transaction Log Basics — Delta records every change in the _delta_log folder and uses periodic checkpoints to avoid replaying the whole log.
Atomicity and Concurrency — The five-step write pipeline ends in an atomic log append, and racing writers are resolved by optimistic concurrency control.
Consistency, Isolation, Durability — Schema and constraint validation enforce consistency, snapshot isolation keeps readers and writers independent, and durability rides on cloud storage.
Append, Copy-on-Write, MERGE — Append only adds files, while update, delete, and merge rewrite affected Parquet files and log the swap in one entry.
Checkpoints and Debugging — Checkpoint compaction, OPTIMIZE, VACUUM, and DESCRIBE HISTORY are presented as fixes for common table problems.
Advanced Delta Features — Liquid clustering, Change Data Feed, dynamic SQL filters, Delta Sharing, and deletion vectors round out the lesson.

PDF notes

Bottom line: You will be able to explain exactly how Delta Lake delivers ACID guarantees on top of plain cloud object storage, and use the transaction log, time travel, and MERGE in your own pipelines. By the end you can trace a write from staged Parquet files to an atomic commit, resolve concurrent write conflicts, and recover a table to an earlier version.

This lesson on Delta Lake: ACID, Time Travel & Merges is hands-on and example-driven. You will be able to explain exactly how Delta Lake delivers ACID guarantees on top of plain cloud object storage, and use the transaction log, time travel, and MERGE in your own pipelines. By the end you can trace a write from staged Parquet files to an atomic commit, resolve concurrent write conflicts, and recover a table to an earlier version.

What You'll Be Able To Do

Explain how Delta Lake delivers each of the four ACID guarantees on cloud object storage
Trace a write through the stage, prepare, validate, commit, and cleanup steps
Read a past table version using versionAsOf and timestampAsOf
Resolve concurrent write conflicts using optimistic concurrency control
Write a MERGE statement that upserts and deletes in a single transaction
Diagnose small-file and transaction-log problems with OPTIMIZE and VACUUM

Detailed Concept Walkthrough

1. ACID Guarantees in Data Lakes

ACID — Atomicity, Consistency, Isolation, Durability — is the set of guarantees that makes concurrent reads and writes on one table safe. Delta Lake's whole selling point is bringing those guarantees to data lakes, which used to be the wild west.

Mechanism: Atomicity means a transaction happens completely or not at all, like a bank transfer where the money either moves or stays put. Consistency means your declared rules — say, no customer can have a negative balance — hold before and after every transaction. Isolation keeps concurrent transactions in their own private bubbles so they never step on each other's toes, and Durability means a committed transaction stays committed even if the server crashes or gets hit by lightning.
Under the Hood: Relational databases like Postgres, MySQL, and SQL Server have implemented these four properties for decades using locks and write-ahead logs. Cloud object storage has no lock manager and no in-place mutation, so Delta Lake had to rebuild the same guarantees on top of immutable files.
Failure Mode Without ACID: Picture a lake with three simultaneous jobs: one writing customer data, one writing transaction records, one reading for analytics. Without ACID, Job A writes corrupted data, Job B overwrites everything, and Job C reads an inconsistent mix — reports are garbage and stakeholders are furious.
Best Practice: Treat ACID as the reason you can run concurrent ETL against the same table at all. With Delta, all three jobs run simultaneously, each protected by isolation, and a failure rolls back cleanly instead of leaving half-written state behind.

-- Consistency is enforced on every write: schema plus constraints
CREATE TABLE customers (
  customer_id INT NOT NULL,
  balance     DECIMAL(10,2)
) USING DELTA;

-- Rejected: violates the declared schema and the not-null rule
INSERT INTO customers VALUES ('abc', -10.00);

Key Takeaway: ACID is the contract that lets many writers and readers share one table without corrupting it.

2. Transaction Log and Checkpoints

Delta Lake never mutates your data files directly. Every change is recorded as a new JSON commit in the _delta_log folder, which acts as the table's single source of truth.

Mechanism: Each transaction creates a new sequentially numbered JSON file — 00000000000000000000.json, 00000000000000000001.json, and so on — containing an array of actions. The main action types are add (this Parquet file joins the table, with path, size, record count, per-column min/max, and partition info), remove (this file leaves the table, with when and why), metadata (column names, types, partitioning scheme, format), and txn (an external system transaction reference).
Under the Hood: The log is a ledger: to know the current state of the table you replay entries in order, exactly like reading a bank account statement from the first transaction. Reading millions of JSON entries on every query would be terrible, so Delta writes a checkpoint — a Parquet snapshot of the complete table state — every 10 commits by default.
Execution Flow: A query reads the latest checkpoint to jump to a known-good state, replays only the JSON commits written since that checkpoint, prunes files using the column min/max statistics stored in the log, and finally reads only the relevant Parquet files. That last step is why queries stay fast even when the table holds millions of files.
Best Practice: Checkpoint compaction combines multiple transaction log files into a single Parquet file for even faster scans. Never hand-edit or delete files inside _delta_log — you would be editing the table's source of truth, not just some metadata.

# Every commit is a new, sequentially numbered JSON file
ls _delta_log/
# 00000000000000000000.json
# 00000000000000000001.json
# 00000000000000000010.checkpoint.parquet   <- full table state snapshot

Key Takeaway: The _delta_log is the table; the Parquet files are just data it points at.

3. Atomic Commits and Concurrency

A Delta write is a five-step pipeline that ends in one atomic append to the transaction log — that single append is what makes the whole operation all-or-nothing.

Execution Flow: Stage writes the new Parquet files to a temporary location; Prepare builds the JSON entry for the transaction log; Validate checks the write against constraints and schema; Commit atomically appends the entry to the log; Cleanup marks the now-obsolete files for deletion.
Under the Hood: The magic moment is the atomic append. Appending a single object to cloud storage is atomic, so either the entire transaction is recorded or none of it is — there is no halfway-committed state. If the job crashes mid-write, the log entry never appears and the abandoned temp Parquet files are simply ignored by the next reader, which is a clean rollback.
Mechanism: When two writers race, Delta uses optimistic concurrency control. Writer A and Writer B both start; A finishes first and commits version 2; B tries to commit version 2, Delta sees that version number already exists, and B automatically retries as version 3. Both writes succeed, ordered, with no data corruption and no lost updates.
Best Practice: Because conflicts are only detected at commit time, keep transactions short and avoid having many jobs rewrite the same partitions at once — retries preserve correctness but still cost wall-clock time.

# The commit is one atomic append to _delta_log — no partial writes
(df.write
   .format("delta")
   .mode("append")
   .save("/mnt/lake/transactions"))

Key Takeaway: Atomicity comes from one atomic log append; conflicts are resolved by retrying at the next version.

4. Snapshot Isolation and Durability

Readers pin a version number and see a frozen snapshot of the table, while writers commit new versions independently — so nobody blocks anybody.

Mechanism: When a read starts, Delta captures the current version from the transaction log, say version 50. A concurrent writer adds and removes files and commits version 51, but your read keeps looking at version 50. You see a snapshot frozen in time, completely unaffected by the concurrent write.
Under the Hood: This is snapshot isolation, and its payoff is that readers never block writers and writers never block readers, with multiple writers able to work simultaneously. The trade-off is that you might read slightly stale data, which is an acceptable price in most data lake scenarios.
Durability: Delta leans on the underlying cloud storage — S3, Azure Blob, or GCS replicate your objects across multiple data centers, with S3 offering eleven nines of durability. Delta adds a layer on top: the transaction log is the source of truth, so even if a Parquet file is corrupted or deleted you still have the log entry referencing it and can detect the problem.
Best Practice: VACUUM cleans up old files but respects a retention period, 30 days by default. Inside that window you can recover with time travel using versionAsOf or timestampAsOf; outside it, the old files are gone for good.

# Pin a frozen snapshot by version number
spark.read.format("delta").option("versionAsOf", 50).load("/mnt/lake/transactions")

# Or travel back by timestamp
spark.read.format("delta").option("timestampAsOf", "2024-05-01").load("/mnt/lake/transactions")

Key Takeaway: Readers see a frozen version, writers commit new ones, and the log outlives any single Parquet file.

5. Write Patterns and MERGE

Append is cheap because it only adds files; update, delete, and merge are copy-on-write because Parquet files are immutable and must be rewritten.

Mechanism: A simple append writes new records to one or more Parquet files and records a single ADD action. The query optimizer immediately sees the new files and no existing file is rewritten, so append is nearly always fast.
Copy-on-Write: Update and delete are trickier because you must remove records from an existing file. Delta reads the existing Parquet file, filters out the deleted rows or applies the updates, writes the modified data to a new Parquet file, then records a REMOVE action for the old file and an ADD action for the new one — both in the same log entry, so the old file disappears and the new one appears atomically.
MERGE: MERGE handles insert, update, and delete in a single operation. Delta identifies which existing records match the source data, rewrites those Parquet files with the updates or deletions, writes new Parquet files for the inserts, and records every file operation in one transaction log entry.
Best Practice: Deletion vectors mark rows as deleted instead of rewriting whole files, making deletes and updates much cheaper. As tables grow, run OPTIMIZE to compact many small files into fewer large ones and keep the transaction log manageable.

-- Upsert and delete in one atomic transaction
MERGE INTO silver.customers AS t
USING staging.customers AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Key Takeaway: Append only adds files; update, delete, and merge rewrite affected files and log the swap atomically.

Topics Covered in Delta Lake: ACID, Time Travel & Merges

Intro and Promise (0:00 - 1:00) — Naveen introduces the lesson and promises a technical, under-the-hood look at how Delta Lake implements ACID.
The Four ACID Properties (1:00 - 3:00) — Atomicity, Consistency, Isolation, and Durability are each explained with a bank-transfer style analogy.
Why Data Lakes Need ACID (3:00 - 5:00) — A horror story of three concurrent ETL jobs corrupting a lake shows what breaks without transactions.
Transaction Log Basics (5:00 - 8:00) — Delta records every change in the _delta_log folder and uses periodic checkpoints to avoid replaying the whole log.
Atomicity and Concurrency (8:00 - 11:00) — The five-step write pipeline ends in an atomic log append, and racing writers are resolved by optimistic concurrency control.
Consistency, Isolation, Durability (11:00 - 14:00) — Schema and constraint validation enforce consistency, snapshot isolation keeps readers and writers independent, and durability rides on cloud storage.
Append, Copy-on-Write, MERGE (14:00 - 17:00) — Append only adds files, while update, delete, and merge rewrite affected Parquet files and log the swap in one entry.
Checkpoints and Debugging (17:00 - 19:00) — Checkpoint compaction, OPTIMIZE, VACUUM, and DESCRIBE HISTORY are presented as fixes for common table problems.
Advanced Delta Features (19:00 - 21:00) — Liquid clustering, Change Data Feed, dynamic SQL filters, Delta Sharing, and deletion vectors round out the lesson.

Spark Lakehouse Cheat Sheet

_delta_log — Transaction log directory holding every commit as JSON
```
ls _delta_log/00000000000000000000.json
```

versionAsOf — Read a specific past version of the table

spark.read.format("delta").option("versionAsOf", 50).load(path)

timestampAsOf — Read table state as of a point in time

spark.read.format("delta").option("timestampAsOf", "2024-05-01").load(path)

DESCRIBE HISTORY — List table versions, operations, and timestamps
```
DESCRIBE HISTORY customers LIMIT 20;
```

MERGE INTO — Upsert, update, and delete in one transaction

MERGE INTO silver.customers AS t
USING staging.customers AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

VACUUM — Delete files older than the retention window
```
VACUUM customers RETAIN 168 HOURS;
```
OPTIMIZE — Compact many small files into fewer large ones
```
OPTIMIZE customers;
```
Checkpoint — Parquet snapshot of full table state every 10 commits
```
ls _delta_log/*.checkpoint.parquet
```

Comparison Table

Operation	What Delta Writes	Cost
Append	New Parquet files plus ADD action	Cheapest, no rewrite
Update / Delete	Rewritten Parquet plus REMOVE and ADD	Rewrites whole affected file
MERGE	Rewritten matched files plus new inserts	One log entry, many file ops
Deletion vectors	Marks rows deleted, no rewrite	Fastest deletes, read-time merge

Common Pitfalls

Mistake: Assuming a failed write leaves partial data in the table. Avoid: check the log — no commit entry means readers ignore orphaned temp files.
Mistake: Running VACUUM with zero retention to reclaim space. Avoid: keep the default 30-day retention so time travel still works.
Mistake: Letting thousands of tiny files accumulate in a table. Avoid: run OPTIMIZE to compact small files into fewer large ones.
Mistake: Expecting readers to see a writer's uncommitted changes. Avoid: snapshot isolation means readers see the version captured when the read started.
Mistake: Treating concurrent write conflicts as data loss. Avoid: Delta retries the losing writer at the next version number automatically.

FAQs

Why is appending to a JSON file considered atomic? Cloud object storage makes a single object PUT atomic, so the entire log entry lands or none of it does. That is why Delta commits by appending one new file rather than editing existing ones.
Do readers block writers in Delta Lake? No. Snapshot isolation means a reader pins a version number and writers commit new versions independently, so readers never block writers and writers never block readers.
What happens if two writers try to commit the same version? Optimistic concurrency control detects that the version number already exists and the losing writer automatically retries at the next version number. Both writes succeed, just ordered.
How far back can I time travel? Any version still inside the retention window — 30 days by default. Once VACUUM removes the old files, those versions are no longer readable.
Where does durability come from if Parquet files can be deleted? Cloud storage replicates objects across data centers, and the transaction log is the source of truth. Even a corrupted Parquet file can be detected because the log still references it.
What is a checkpoint actually for? It is a Parquet snapshot of the complete table state written every 10 commits, so a query reads one checkpoint plus a few recent JSON commits instead of replaying the whole log.

Common pitfalls

Mistake: Assuming a failed write leaves partial data in the table. Avoid: check the log — no commit entry means readers ignore orphaned temp files.
Mistake: Running VACUUM with zero retention to reclaim space. Avoid: keep the default 30-day retention so time travel still works.
Mistake: Letting thousands of tiny files accumulate in a table. Avoid: run OPTIMIZE to compact small files into fewer large ones.
Mistake: Expecting readers to see a writer's uncommitted changes. Avoid: snapshot isolation means readers see the version captured when the read started.
Mistake: Treating concurrent write conflicts as data loss. Avoid: Delta retries the losing writer at the next version number automatically.

Hello everyone, my name is Naveen Malik and welcome back to my channel, Data Architect Diaries. If you are building a data platform or working with data lakes, you have probably heard the terms ACID transactions thrown around. But here is the thing, most people use the term without really understanding what's happening under the hood, okay? So, and that's exactly what we are diving into today. We are going to break down how Delta Lake implements ACID transaction at a technical level, and I promise to make it interesting. No boring textbooks here, just real-world insights. So, if you are new here, don't forget to hit the subscribe button and ring the notification bell. You will get weekly deep dives into data architecture topics. So, let's get started. So, all right, let's start with the basics. ACID stands for Atomicity, Consistency, Isolation, and Durability, okay? Think of it like the golden rules of reliable data transactions, okay? So, first we'll talk about the Atomicity. Imagine you are transferring money from one bank account to another, either the money move completely or it does not move at all. So, there is no in between, no account left with missing money. That is the atomicity, all or nothing, okay? Second one is the Consistency. This means your data follow the rule you have set. If you say no customer can have a negative balance, then that rule holds true before and after every transaction, no exceptions. Third one is the Isolation. Let's picture this, you have two people try to read and update the same data simultaneously, okay? Isolation makes sure they don't step on each other toes. One process does not interfere with another. Each transaction happens like it is in own private bubble, okay? Fourth one is the Durability. Once you have committed transaction, it stay committed. Even if your server crash, burns down, or gets hits by lightning, that data is safe and retrievable, okay? So, now here is what it gets interesting. Traditional database like Postgres, SQL or MySQL have implemented these features for decades, but Delta Lake brought this reliability to data lakes, which is a pretty big deal because data lakes used to be the wild west. So, why do we even care about ACID in data lakes? Let me tell you a horror story, okay? You have got a data lake storing millions of transactions. Multiple ETL jobs are running simultaneously. One job is writing customer data, another is writing transaction records, a third is reading from the same location to power analytics. So, without ACID, Job A write corrupted data, Job B overwrites everything, Job C read inconsistent data. So, your reports are garbage, your stakeholders are furious, and you are updating your resume. But with the ACID Delta Lakes, that what will be happen, all three operations can happen simultaneously, each one is protected by isolation. If something fail, it roll backs cleanly. Your data stay consistent. Your stakeholders are happy, and you keep your job. This is exactly why Databricks created Delta Lake. They wanted to bring enterprise-grade reliability to the data lake world, and the way they did it is actually pretty clever. So, let's talk about how Delta Lake actually works at the architectural level, okay? Delta Lake sits on the top of cloud object storage like S3, Azure Blob storage, or GCS. The key innovation is the transaction log, also called the commit log. Here is the basics idea. Instead of directly modifying files in your data lake, which would be chaotic, every change to a table gets recorded in a _delta_log folder. This log acts like a journal or a ledger, okay? Think of it like as a bank accounting ledger. Every transaction get written down in order. If you want to know the current state of an account, you read through all the transaction from the start. The same concept applied here. So, the transaction log contains JSON files that record every single change. What files were added, what files were removed, what metadata changed, when it happened, and who did it. This might sound inefficient, and honestly, reading through millions of log entries every time would be terrible. So, Delta Lake also maintain checkpoints, periodic snapshots that say at the and at this point in time, here is exactly what the table looks like, okay? So, instead of reading from transaction zero, you jump to latest checkpoint and read only recent transaction, that will be the much faster, okay? So, let me break down the transaction log in more detail because this is the heart of Delta ACID implementation, okay? So, every transaction in Delta create a new file in the _delta_log directory. These files are numbered sequentially in binary format, 000.json and 0001.json, okay? And so on. Each file contains a JSON array of transaction. The main types of transaction are: Add file: I am adding this Parquet file to the table, it includes the file path, file size, number of records, min or maximum values for each column, and the partition information. Second one come into the remove file: I am removing this file from the table, it include which files to remove, when it was removed, why we need to be removed for this one. Third one come into the metadata: Here is the information about the table itself. This include column names and types, partitioning scheme, and format information. Third one come into the transaction: Here is an external system transaction reference, this is useful when Delta integrate with external system. So, now here is where it gets powerful. When a query run in Delta Lake, it it reads the latest checkpoint, that will jump to a known good state. Second one, read all transaction log files since the checkpoint update its view, okay? Third one, filter files based on the query. It will use column statistics for pruning. The last one, read only relevant Parquet file that will use for the lazy loading. So, this design ensure that even with millions of files, queries are still fast because they use statistics to skip reading than the unnecessary files, okay? All right, let's get into the meat of how ACID actually work in Delta Lake, starting with atomicity, all-or-nothing execution. When you execute a write operation in Delta Lake like insert, update, delete, or merge, here is what happens behind the scene: First one is the stage: New Parquet file are written to a temporary location. Second one for prepare transaction: Delta prepare a JSON entry for the transaction log. Third one is the validate: It checks that the write is valid and would not violate constraints. Fourth one is commit: It atomicity appends to the transaction log. Fifth one is clean up: Old files are marked for deletion. So, the key word here is atomicity appends. This is the magic moment because we are appending to a JSON file, which is an atomic operation on cloud storage. Either the entire transaction get recorded or it does not. So, there is no partial writes, there is no halfway committed state. Now, what happen if something fails? Let's say your write job crash halfway through. The transaction log entry was never written, the temporary Parquet file sit abandoned. So, next time someone run a query, those temp files are ignored. It's like the failed transaction never happened, clean and it will be the clean roll back, okay? So, what if two write operation try to commit simultaneously? Delta use optimistic concurrency control. So, here is how it works: Writer A start writings, Writer B start writing. When A finish first and commits version 2, then B finish and try to commit version 2. So, Delta see the conflict, Writer B version number already exist, so Writer B automatically retries with version 3, and then Writer B successfully committed version 3. So, both write succeed, but they are ordered. No data corruption, no lost updates, okay? Now moving on consistency and isolation. In Delta Lake, consistency is enforced through constraint and schema validation, okay? When you define a table with specific column types and rules, Delta enforce them throughout, okay? Every write operation validate against the schema. If you try to insert a string into an integer column, Delta rejects it, okay? So, if you define a not-null constraint, Delta enforce it. But here is the deeper layer: because every change is recorded in the transaction log, you can always validate that each state transition is valid. It's like having a mathematical proof of correctness, okay? So, now isolation isolation is where it gets really stri- interesting. Delta implements snapshot isolation also. That here is the concept: when you start a read operation, Delta capture the current version number from the transaction log. Let's say you are at version 50, okay? Meanwhile, some other writer is modifying the table, they add file, remove file, and commit version 51, okay? So, but your read operation, it is still looking at version 50. You see a snapshot frozen in time, you are not affected by concurrent writes. This is a powerful because readers never block writers, writers never block readers, and multiple writers can work simultaneously. So, everyone sees a consistent view. The drawback is that you might read data that is that is slightly stale, but that's an but that is an acceptable trade-off in most data lake scenario, okay? Hope you have understood for the isolation. Finally, durability: once you commit, is stay committed. So, Delta durability rely on the underlying cloud storage durability. So, when you write to S3, Azure, or GCS, this service replicate your data across multiple data centers, okay? They provide to 99.99% durability, that means 11 9s for S3, okay? But Delta add an extra layer on top. The transaction log is the source of truth. Even if a Parquet file gets corrupted or deleted, you have the transaction log entry that reference it, okay? So, you can detect corruption and potentially recover. Also, Delta Lake include a vacuum operation that clean up old file, but it keeps a retention period, default is the 30 days. But so if you accidentally deleted sometimes something or need to recover to a previous version, you can use time travel to go back to any version within the retention window, okay? You can also use like as a timestamp as of or you can also, if you have know about the version number, you can also version as of, okay? Now let's deep deeper into how Delta handles specific write patterns, okay? So, when you do a simple append operation, what will happen? New records are written to one or more Parquet file, okay? A transaction log entry records the ADD file action, query optimizer immediately see these new files, no rewriting of existing files, okay? So, this is super-efficient, append is nearly always fast. But what about the when you update or delete operation? This is trickier because you need to remove record from existing file. So, then Delta use strategy called copy-on-write, okay? So, what will happen? First, it read the existing Parquet file, filter out the records you are deleting or update the ones you are modifying, write the modified Delta to a new Parquet file, and record the old file as removed, that is a remove file action, and then record the new file as added, that means the add file action, okay? So, both actions are in the same transaction log entry, so automatically the old file disappear and the new file appears, okay? Here we have also merge, which is a very common operation in Delta warehouse, okay, data warehouse for that one, okay? So, this is incredible powerful because it handles insert, update, and delete in a single operation behind the scene. So, Delta identify which existing record match the source data, rewrite those Parquet file with update or deletions, write new Parquet file for inserts, and records all file operation in a single transaction log entry, okay? So, everything happens automatically. Now here is the potential problem: as your table grow, the transaction log accumulates thousands of millions or entries. Reading through all of them would be slow, okay? So, that's is where checkpoints come in. So, every 10 transaction by default, Delta creates a checkpoint file. This checkpoint is a Parquet file that contains the complete state of the table at that point, okay? So, instead of reading through 10 transactions log entries, you jump to the checkpoint and read it once. Here is the like checkpoint file structure, you can also JSON file will be there, okay? So, when query, latest checkpoint is transaction 10, read checkpoint Parquet and like contains all add file, remove file, or metadata from transaction 0 to 10. After then, it will also read the transaction from 11 to 16, and after that, it will combine for complete pictures. This is a way faster than reading 17 individual uh JSON files, okay? So, there is an also an optimization called checkpoint compaction, where can combine multiple transaction log file into a single Parquet file for even faster scans. There we can also debugging some common issue, like we when we have the too many small files, okay? The on that time, you can also use uh OPTIMIZE command to compact those small file to a big one, okay? Second one is the large transaction log. For this, you need to run the VACUUM command to remove old file, and then you can also the OPTIMIZE. Third one, that is the write conflicts. So, on that one, you can also for the increase isolation level, use row-level versioning, and after that, you can also redesign write, okay? Fourth one is fourth one is the corrupted state. So, what we'll do? You can also use the timestamp as of if you have the corrupted to pin a specific version, okay? And after then, if you need to have the diagnostic commands is there, like SELECT * FROM <table_name>.history, you can also LIMIT 20, so you can also view all version, okay? And like you can other one, the timestamp as of that you can also use for the time travel for this one, okay? Hope you have understood the common issue. Here in Delta, we are also having some advanced features, okay? These are the liquid clustering: instead of strict partitioning, Delta can be use liquid clustering for better distribution, okay? More flexible than partitioning for highly skewed data. It can be asked from into from the interviewer side that what is the liquid cluster, okay? Second one is the CDF, Change Data Feed. So, automatically capture what changed, which row were added, updated, or deleted, useful for CDC pipeline downstream, okay? Third one is the dynamic SQL filters. It will also push down filter even complex query, reduce amount of data scan, okay? Fourth one is the Delta Sharing. It will for the share Delta table across organization securely. It also maintain ACID guarantees across organization boundaries, okay? Fifth one is the deletion vectors. Instead of rewriting files to remove records, mark row as a deleted, okay? Will You can also make delete and update operation, that you can also much faster for this one, okay? Last one is the Iceberg compatibility, okay? Delta is working towards compatibility with Apache Iceberg format, so you will also allow multi-engine environment for this one. So, the future is bright for Delta Lake. The engineering team is constantly improving performance and adding features while maintaining ACID guarantees, okay? So, finally, uh, you are a lot of information, let recap. One first one is the transaction log, okay? Act as a reliable ledger of all changes. Second one is the atomicity is guaranteed through atomic appends of the log. Third one is the consistency is enforced through schema validation. Fourth one is the isolation, that is for the implemented via snapshot isolation. Fifth one is the durability, relying on cloud storage plus transaction log, okay? Third, that we have also talked about the checkpoint and also discuss about the time travel, okay? So, if you found this deep dive helpful, please give this video a thumbs up, leave a comment below and tell me what topic you would like me to dig into next. Subscribe to the channel for the more deep dive into data architecture. And if you are using Delta Lake in production, drop a comment about your use case. I would like to love hear about it. So, thanks for watching, and I will see you in the next episode for the Data Architect Diaries with Naveen Malik. Thank you.

Frequently asked questions

Why is appending to a JSON file considered atomic?

Cloud object storage makes a single object PUT atomic, so the entire log entry lands or none of it does. That is why Delta commits by appending one new file rather than editing existing ones.

Do readers block writers in Delta Lake?

No. Snapshot isolation means a reader pins a version number and writers commit new versions independently, so readers never block writers and writers never block readers.

What happens if two writers try to commit the same version?

Optimistic concurrency control detects that the version number already exists and the losing writer automatically retries at the next version number. Both writes succeed, just ordered.

How far back can I time travel?

Any version still inside the retention window — 30 days by default. Once VACUUM removes the old files, those versions are no longer readable.

Where does durability come from if Parquet files can be deleted?

Cloud storage replicates objects across data centers, and the transaction log is the source of truth. Even a corrupted Parquet file can be detected because the log still references it.

What is a checkpoint actually for?

It is a Parquet snapshot of the complete table state written every 10 commits, so a query reads one checkpoint plus a few recent JSON commits instead of replaying the whole log.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.