Course Wiki
dlt
The library-first data ingestion tool taught in the Data Engineering and LLM Zoomcamp ingestion workshops.
Related Pages
dlt (data load tool) is a Python library for building ingestion pipelines: declare a source (REST API, filesystem, database), and dlt handles API pagination, normalization of nested JSON into tables, incremental loading with tracked cursors, and schema inference as it loads into a destination such as DuckDB or BigQuery.
Data Engineering Zoomcamp’s workshop scales API reading and teaches incremental loading with it. LLM Zoomcamp’s workshop uses the same library to ingest LLM application traces — filesystem and REST API sources into DuckDB — and then analyzes them in marimo notebooks. One library, the same declarative pipeline shape, two problem domains.
Taught in
- Data Engineering Zoomcamp — Workshop: Data Ingestion
- LLM Zoomcamp — Workshop: Data Ingestion