Apache Spark Streaming

Apache Spark Streaming, in its current form Spark Structured Streaming, is the streaming component of the Apache Spark project. Its defining design decision is that it is not a separate system: streaming applications and pipelines are built with the same familiar Spark APIs a team already uses for batch work.

That choice resolves a problem that has shaped this whole category. Organisations running both batch and streaming workloads have historically maintained two technology stacks, with two sets of skills, two deployment paths and two implementations of the same business logic. Spark Structured Streaming provides the same structured APIs, DataFrames and Datasets, across both, so neither the code nor the expertise has to be duplicated. A direct consequence is migration: existing batch Spark jobs can be moved to streaming jobs through the unified API rather than rewritten.

The second design decision concerns complexity. Structured Streaming abstracts away concepts that stream processing normally forces developers to handle explicitly, including incremental processing, checkpointing and watermarks. A developer can build a streaming pipeline without first learning those concepts or the tools that manage them, while the engine handles correctness underneath.

Apache Spark is open source under the Apache Software Foundation, so there is no licence cost. It is available directly, and also underpins managed offerings from cloud providers and vendors for teams that would rather not operate a cluster themselves. Because it is a component of Spark rather than a standalone product, adopting it means adopting Spark, which is worth weighing if streaming is the only workload you have.