Apache Spark

Apache Spark is a unified engine for large-scale data analytics, described by the project as a multi-language engine for executing data engineering, data science and machine learning on single-node machines or clusters, with Spark Connect clients published for Go, Rust and Swift alongside a Kubernetes operator and Docker images.

The unified engine framing is the substance of why Spark displaced what came before it, and it is worth understanding what it replaced.

Hadoop's original processing model required expressing every computation as a map and reduce pair, which worked for straightforward batch transformations and became painful for anything iterative. Machine learning training, graph algorithms and interactive exploration all involve passing over the same data repeatedly, and each pass meant writing intermediate results to disk and reading them back. That made a class of work impractically slow rather than merely inconvenient.

Spark kept data in memory between operations, which changed those workloads by orders of magnitude and made interactive analysis on large datasets possible for the first time.

The multi-language support is more consequential than a feature list suggests. Data engineers generally work in Scala or Java, data scientists in Python or R, and analysts in SQL. An engine addressable from all of them means one processing platform rather than a translation layer between teams, and it removes the common failure where a model developed in Python has to be rewritten by an engineer for production.

Running on a single node as well as a cluster matters for development practice, since the same code can be written and tested on a laptop before running against the full dataset.

The Kubernetes operator reflects where deployment has moved, with Spark increasingly run on general container infrastructure rather than on a dedicated Hadoop cluster.

Spark is free and open source under the Apache licence, with managed offerings from every major cloud provider.