Course Wiki

Workflow Orchestration

Coordinating multi-step data and ML workflows: schedules, dependencies, retries, and observability.

Workflow orchestration is the conductor role for pipelines: multiple tools and steps need to run in order, on a schedule or in reaction to events, with errors monitored and handled. Data Engineering Zoomcamp introduces the concept with Kestra; MLOps Zoomcamp applies it to ML — turning a training notebook into a Python script, then wrapping the script in an orchestrator such as Prefect, Airflow, Dagster, Kestra, or Mage to make it a production pipeline.

The shared lesson across the three courses: anything that must run repeatedly and reliably belongs to an orchestrator, which handles scheduling, dependency ordering, retries, and run history so the pipeline code does not have to.

Taught in

• LLM Zoomcamp

  • Module 1: Agentic RAG
    • Other Frameworks
  • Module 3: Orchestration
    • AI Orchestration
    • AI Agents

• AI Dev Tools Zoomcamp

  • Module 5: Coding Agent Building Blocks: Reusable Skills and Specialized Subagents
    • Module 5 — Coding Agent Capabilities: MCP, Skills, Plugins, and Custom Agents

• MLOps Zoomcamp

  • Module 3: Orchestration & ML Pipelines
    • Using an Orchestrator
    • Homework
  • Module 07: Course Project
    • Course Project

• Data Engineering Zoomcamp

  • Module 2: Workflow Orchestration
    • 2.1.1 - What is Workflow Orchestration?
  • Module 5: Data Platforms
    • Introduction to Bruin ## Related concepts

• Kestra

• Model Deployment

• dlt

• MLflow

• BigQuery