Apache Hadoop

Apache Hadoop is an Apache Software Foundation project developing open source software for reliable, scalable, distributed computing, published across maintained release lines including 3.4.0, 3.3.6 and 2.10.2, with a wiki, community bylaws, criteria for committership, mailing lists and published security advisories.

Hadoop's historical significance is that it made distributed data processing accessible on commodity hardware, and the ideas it established remain the foundation of the field even as the specific implementation has been displaced for many workloads.

The core insight was to move computation to the data rather than data to the computation. When a dataset is too large for one machine, the conventional approach of loading it into a processing system fails on the transfer alone. Hadoop distributed the data across many machines and ran the processing on the machine holding each piece, which is why it scaled where previous approaches could not.

An honest assessment of where it now sits is more useful than a feature list. Cloud object storage and separated compute engines have replaced Hadoop for a great many workloads that would once have required it, because storing data in object storage and running a query engine against it is simpler and cheaper than operating a cluster where storage and compute are bound together.

Where Hadoop remains relevant is substantial nonetheless. Large existing deployments represent significant investment and continue to run. Organisations that cannot use public cloud for regulatory or sovereignty reasons still need on-premise distributed processing. And the ecosystem that grew around Hadoop, including formats and schedulers, remains in wide use well beyond Hadoop clusters themselves.

Published CVEs and a formal security process are worth noting for anyone running it, since a distributed system holding an organisation's data warrants the same patching discipline as any other infrastructure, and older release lines eventually stop receiving fixes.

The software is free and open source under the Apache licence, with commercial distributions and support available from third parties.