Delta Lake

Delta Lake is an open source storage framework enabling a format-agnostic lakehouse, working with compute engines including Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric. Through UniForm, Delta tables can be read by Iceberg and Hudi clients, and the project's aim is one format unifying ETL, warehousing and machine learning in a lakehouse.

What it adds to a data lake is transactional guarantees, which is the distinction the whole lakehouse idea rests on. Files in object storage have no notion of a transaction, so a reader can see a half-written update and a failed job can leave a table corrupt, and a transaction log over those files restores the behaviour people expect from a database.

Time travel follows from the same log and is more useful than it first appears. Querying a table as it stood last Tuesday answers questions about when a number changed that are otherwise unanswerable, and it makes reproducing a model training run possible.

Interoperability with Iceberg and Hudi clients is the strategically significant piece. The lakehouse format contest has been costly for users, and a table readable by all three removes the need to pick a winner before committing data to a format.