Course Wiki

Spark

Distributed batch processing: DataFrames, Spark SQL, and the internals of groupBy and joins.

Module 6 moves from single-machine analytics to distributed batch processing with Apache Spark. Spark splits large datasets and the computation over them across a cluster; the module works with the DataFrame API and Spark SQL, which look like pandas and SQL but execute across many machines.

The distinctive part of the module is internals: how groupBy works (shuffling rows to group keys across the cluster) and how joins distribute and shuffle data — the knowledge that separates pipelines that run from pipelines that OOM or crawl. Batch itself is the processing model: data is collected first and processed in chunks, in contrast to the streaming module that follows.

Taught in

• Data Engineering Zoomcamp

  • Module 6: Batch Processing
    • 6.1.2 Introduction to Spark
    • 6.3.1 First Look at Spark/PySpark
    • 6.3.2 Spark Dataframes
    • 6.3.4 SQL with Spark
    • 6.4.1 Anatomy of a Spark Cluster
    • 6.4.2 GroupBy in Spark
    • 6.4.3 Joins in Spark
    • 6.5.1 Operations on Spark RDDs
    • 6.5.2 Spark RDD mapPartition
    • 6.6.2 Creating a Local Spark Cluster
    • 6.6.4 Connecting Spark to Big Query ## Related concepts

• Kafka

• Stream Processing

• BigQuery

• Partitioning and Clustering

• Workflow Orchestration