Course Wiki

Kafka

The distributed event log behind streaming pipelines: topics, producers, consumers, and schemas.

Kafka is a distributed log built for streaming: producers append events to topics, consumers read them at their own pace, and the broker keeps the events durable and ordered per partition. The module’s theory track teaches the concepts — topics, partitions, consumer groups, delivery semantics — with Java code examples.

Schema management with Avro rounds out the theory: producers declare message schemas, and the schema registry validates compatibility as data evolves, so consumers do not break when a field changes. The workshop track then builds a real-time pipeline with PyFlink and Redpanda (a Kafka-compatible broker) writing into PostgreSQL.

Taught in

• LLM Zoomcamp

  • Module 2: Vector Search
    • Vector Search with sqlitesearch
  • Module 5: Monitoring
    • Next Steps

• Data Engineering Zoomcamp

  • Module 7: Streaming
    • Streaming ## Related concepts

• Stream Processing

• Spark

• dlt

• Workflow Orchestration