Yelp has built an internal Training Orchestrator to replace the separate Spark training scripts that individual machine-learning teams had been maintaining. InfoQ reports that the new framework uses configuration files and a directed acyclic graph, or DAG, to define and run training pipelines.
The old setup tightly coupled model training to Spark clusters and job launchers. That made local runs difficult, slowed iteration, and scattered validation logic across custom scripts. A small change often required submitting a cluster job before developers could see whether it failed.
Training Orchestrator separates what should run from how it runs. Pipelines are described with Pydantic-based configuration objects, validated before compute is used, and executed in dependency order. A shared Spark and MLflow context lets the same step definitions run locally, in notebooks, or in production.
The change is not a new consumer product, but it reflects a common maturity step in enterprise AI: once many teams train models, platform consistency can matter as much as model architecture.