Delta Lake

This page was inspired by my Apache Iceberg explorations: same problem (turning a bucket of Parquet into a real table), different design. For now I’ve done way less digging into the Delta Lake format than into Iceberg.

Delta Lake Explorer

Delta Lake Explorer is an interactive visualization I wrote to make Delta Lake’s internals tangible. You can simulate operations like append, delete, update, OPTIMIZE and VACUUM to see how the transaction log grows and what files are created “under the hood”, and you can also time-travel back through versions.

It has three levels of detail that show more and more concepts.

• Try it live ->
• Source on GitHub ->

Notes from my Delta Lake learnings

• Delta Lake is a table format (not a file format, not an engine, not a service), like Iceberg
• the data files are Parquet and, as in Iceberg, every file is immutable
• the heart of it is the transaction log: a _delta_log/ directory sitting next to the data
• where Iceberg builds a tree of metadata files, Delta keeps a flat, ordered sequence of commits: 000...000.json, 000...001.json, … each one an atomic version of the table
• each commit file is a list of actions: add a data file, remove a data file, change metaData, bump the protocol, record commitInfo
• the current state of the table is just “replay the log and apply every add/remove” - the highest-numbered log file is the current version, no external pointer needed
• atomic commits work by atomically creating the next numbered log file: if two writers both try to create version N, only one can win
• that mutual-exclusion guarantee is trivial on a real filesystem but not on all object stores, so S3 multi-cluster writes need a coordinating LogStore (historically DynamoDB) to arbitrate who gets version N
• concurrent writers use optimistic concurrency: build your commit assuming nothing changed, and if someone else committed first, re-read the log and retry
• reads are fast because each add action carries per-file stats (min/max, null counts) inline in the log, so engines can skip files before opening them - the same idea as Iceberg’s manifests, just stored differently
• checkpoints keep log replay cheap: every 10 commits (by default) Delta writes a Parquet checkpoint summarizing the whole table state, plus a _last_checkpoint file pointing at it, so readers start from the checkpoint instead of version 0
• time travel is free: query by version number or by timestamp, since old data files stay on disk until cleaned up
• VACUUM is what reclaims space by deleting files no longer referenced, past a retention window (7 days by default) - which is also what bounds how far back you can time-travel
• schema evolution and schema enforcement are both first-class (add columns, or reject writes that don’t match)
• the whole thing came out of Databricks and open-sourced under the Linux Foundation

Iceberg vs. Delta

The two formats converge on the same guarantees (ACID, time travel, schema evolution, Parquet data) but get there differently:

Other cool features

Physical partitioning, then liquid clustering

Classic Delta partitioning is Hive-style and physical: partition by a column and Delta lays the data out in col=value/ directories. It works, but it inherits the old Hive problems - you can over-partition into millions of tiny files, and you can’t change the partitioning without rewriting data.

Liquid clustering is the newer answer. Instead of fixed directories you declare clustering keys, and Delta clusters the data on those keys as it writes and compacts - no directory explosion, and you can change the keys later without rewriting everything.

Deletion vectors (merge-on-read)

Data files are immutable, so deleting or updating a row has the same copy-on-write vs. merge-on-read tradeoff as Iceberg:

• Copy-on-write (default): rewrite the Parquet file(s) containing the affected rows, minus the changed ones. Cheap reads, expensive writes.
• Merge-on-read via deletion vectors: leave the data file untouched and write a compact bitmap marking which rows are gone. Cheap writes, and the engine applies the vector at read time. OPTIMIZE later folds the vectors back into rewritten files.

Deletion vectors are essentially the same idea Iceberg is moving to in its v3 spec.

Protocol versions and table features

Every Delta table declares a protocol with a minimum reader and writer version, so an old engine won’t silently mis-read a table that uses features it doesn’t understand.

The monotonic version numbers got coarse (each bump dragged in several features at once), so newer Delta uses table features: a table lists exactly which named reader/writer capabilities it needs (deletion vectors, column mapping, etc.), and an engine can open it as long as it supports that specific set. More granular capability negotiation than a single version number.

Other notes

• apparently there is now “UniForm” which makes Delta write Iceberg metadata alongside its own -> an Iceberg engine can read the same table
• MERGE INTO is the upsert primitive (atomic commit); useful for CDC