Course Wiki

Data Engineering Zoomcamp

DataTalks.Club's free nine-week data engineering course: build an end-to-end pipeline with Docker, Terraform, Kestra, BigQuery, dbt, Spark, and Kafka.

Data Engineering Zoomcamp is DataTalks.Club’s free nine-week course on data engineering fundamentals. Learners build an end-to-end data pipeline from scratch — ingestion, warehousing, transformation, batch, and streaming — using industry-standard tools. No prior data engineering experience is required; basic coding and SQL familiarity are.

All materials are open source in the course repository, with videos on YouTube and a course FAQ. The live cohort starts each January; everything is also available self-paced.

Curriculum

All modules, each with its lesson-level notes:

• Module 1: Containerization and Infrastructure as Code
• Module 2: Workflow Orchestration
• Module 3: Data Warehousing
• Module 4: Analytics Engineering
• Module 5: Data Platforms
• Module 6: Batch Processing

• Module 7: Streaming

• Module 1: Containerization and Infrastructure as Code — GCP basics, Docker and Docker Compose, PostgreSQL in Docker, and Terraform for infrastructure setup.
• Module 2: Workflow Orchestration — data lakes and workflow orchestration with Kestra: scheduled and event-driven pipelines defined as YAML.
• Workshop: Data Ingestion — scalable API reading, normalization, and incremental loading with dlt.
• Module 3: Data Warehousing — BigQuery, partitioning and clustering, query best practices, and machine learning in BigQuery.
• Module 4: Analytics Engineering — data modeling with dbt on DuckDB and BigQuery, plus testing, documentation, and deployment (analytics engineering).
• Module 5: Data Platforms — end-to-end pipelines with Bruin: ingestion, transformation, orchestration, quality, and metadata in one platform.
• Module 6: Batch Processing — Apache Spark, DataFrames and Spark SQL, and the internals of groupBy and joins.
• Module 7: Stream Processing — Kafka concepts and a PyFlink workshop building a real-time pipeline with Redpanda, Flink, and PostgreSQL (stream processing).
• Final project — a real-world pipeline with peer review.

Looking for a specific concept? The Concept Glossary links every concept this course teaches to the exact modules and lessons that cover it.

Who it is for

The course targets developers, analysts, and data scientists moving into data engineering. The prerequisite bar is deliberately low — basic coding experience and familiarity with SQL, with Python helpful but optional.

Projects and certificate

Certificates require the final project plus peer reviews during a live cohort. The final project — problem statement, cloud infrastructure, ingested dataset, warehouse or lake, transformations, dashboard, and tests — is a complete portfolio artifact.