Apache Hive

Apache Hive is a distributed, fault-tolerant data warehouse system enabling analytics at massive scale, published by the Apache Software Foundation with a Docker quickstart, design documents, a developer FAQ and an open contribution process.

Hive's historical contribution was to make distributed data processing accessible to people who write SQL, and that mattered more than any technical property it had.

The situation it addressed was stark. Early Hadoop processing required writing Java programs expressing computation as map and reduce operations. That was tractable for software engineers and completely inaccessible to the analysts, statisticians and business users who actually had the questions worth asking. An organisation could store enormous quantities of data and have perhaps three people capable of querying it, all of them engineers with a queue of other work.

Hive introduced a SQL interface that compiled to those distributed jobs underneath. Suddenly anyone who could write a query could interrogate data at a scale that had previously required a specialist, and the bottleneck moved from access to interpretation, which is a far better place for it.

The metastore, which catalogues what tables exist and where their data lives, turned out to be Hive's most durable contribution. Long after query engines moved on, the Hive metastore remained the standard catalogue that other engines read, which is why the name persists in architectures that no longer run Hive queries at all.

An honest assessment of its current position is useful: newer engines including Trino, Spark SQL and Presto generally outperform Hive for interactive querying, and the modern equivalent is usually a query engine over an open table format. Hive remains in wide production use in established deployments, and its metastore is still ubiquitous.

Hive is free and open source under the Apache licence.