Apache Airflow

Apache Airflow is a platform created by its community to programmatically author, schedule and monitor workflows, and it has become the default choice for data-engineering pipeline orchestration specifically, distinct from the IT-operations focus of most other tools in this category. Pipelines are defined as Python code rather than through a GUI or proprietary job language, which is precisely why data teams gravitated toward it.

Its four stated principles are scalable (a modular, message-queue-based architecture that scales to an arbitrary number of workers), dynamic (pipelines defined in Python enabling dynamic generation), extensible (custom operators and extended libraries), and elegant (lean, explicit pipelines using Jinja templating). Airflow is free and open source, maintained by the Apache community, with an annual Airflow Summit. Several vendors in this category, including Broadcom's Automic, explicitly build integration layers for Airflow rather than treating it as a competitor to route around.