Course Wiki

Kestra

The open-source, event-driven orchestrator Data Engineering and LLM Zoomcamps use to define pipelines as YAML.

Kestra is the orchestration tool taught in both Data Engineering Zoomcamp and LLM Zoomcamp. Workflows are declared in YAML as tasks with dependencies: extraction tasks feed transformation tasks, everything runs on schedules or on event triggers, and retries, error handling, and logging are part of the definition rather than custom scripts.

Data Engineering Zoomcamp builds ETL and ELT pipelines against GCP with it; LLM Zoomcamp uses the same engine to orchestrate LLM application workflows. The transferable idea: a pipeline is configuration, reproducible and versionable, not a pile of cron scripts.

Taught in

• LLM Zoomcamp

  • Module 3: Orchestration
    • AI Orchestration
    • Context Engineering
    • Setting up Kestra
    • AI Copilot
    • Retrieval Augmented Generation
    • AI Agents
    • Multi-Agent Systems
    • Best Practices
    • Next Steps

• MLOps Zoomcamp

  • Module 3: Orchestration & ML Pipelines
    • Using an Orchestrator

• Data Engineering Zoomcamp

  • Module 2: Workflow Orchestration
    • 2.1.2 - What is Kestra?
    • 2.2.1 - Installing Kestra
    • 2.2.2 - Kestra Concepts
    • 2.4.3 - Create an ETL Pipeline with GCS and BigQuery in Kestra
    • 2.5.3 - AI Copilot in Kestra ## Related concepts

• Workflow Orchestration

• dlt

• Terraform

• BigQuery

• Docker