Apache Iceberg

Apache Iceberg is an open table format for analytic datasets, described by the project as a high-performance format for huge analytic tables that brings the reliability and simplicity of SQL tables to big data, while making it possible for engines including Spark, Trino, Flink, Presto, Hive and Impala to work on the same tables at the same time safely.

That last capability is the reason Iceberg has been adopted so widely, and it solves a problem that quietly constrained data platforms for a decade.

Before table formats like this, data in a lake was a collection of files in object storage with a directory convention layered over it. Every engine had to agree on that convention, and none of them could safely write while another was reading, because there was no notion of a transaction. The practical consequences were familiar to anyone who ran such a platform: a query returning half-written results because a job was still writing, no way to change a schema without rewriting everything, and no way to know what the table looked like an hour ago.

Iceberg introduces a proper table abstraction with atomic commits, so a reader always sees a consistent snapshot and writers do not corrupt readers. Schema evolution becomes a metadata operation rather than a full rewrite, and time travel to a previous snapshot becomes possible because the snapshots are tracked rather than overwritten.

Multi-engine access is the strategic point. An organisation can run Spark for its pipelines, Trino for interactive queries and Flink for streaming against the same tables, without copying data between systems or maintaining three versions of the same dataset. That was previously the main source of both cost and disagreement between teams.

The format is also what makes storage genuinely separable from compute, which is the architecture that has displaced traditional Hadoop clusters for most new work.

Iceberg is free and open source under the Apache licence, with managed implementations offered by cloud and platform vendors.