Local PySpark Setup & First DataFrame Job

You do not need a cluster for this course. A laptop with Python 3.10+ and 8 GB RAM runs every lab. We use local-mode Spark (master("local[*]")).

1. Install (5 minutes)

pip install "pyspark==3.5.*" pandas pyarrow delta-spark
java -version   # needs Java 8/11/17 on PATH; Spark fails without it

Verify:

from pyspark.sql import SparkSession
spark = (SparkSession.builder.appName("smoke")
         .master("local[*]")
         .config("spark.sql.shuffle.partitions", "4")  # small local shuffle
         .getOrCreate())
print(spark.version)

2. The 4 DataFrame rules

Transformations are lazy — filter/select/join build a plan; nothing runs until an action (show/count/write/parquet) triggers it.
Read the plan, not the code — df.explain() shows whether Spark will scan, shuffle, or broadcast. A lab that cannot explain its plan is incomplete.
Narrow vs wide transforms — filter/select/withColumn need no shuffle; groupBy/join/distinct shuffle data across partitions and dominate runtime.
Partitions are parallelism — each partition is one task. Too few (1-2) = idle cores; thousands of tiny files = driver overhead. Repartition before writes.

3. First job: CSV -> partitioned Parquet

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
df = spark.read.option("header", True).csv("data/orders.csv")
(df.filter("status = 'completed'")
   .repartition(4, "order_date")
   .write.mode("overwrite")  # dynamic mode: only touched order_date= partitions
   .partitionBy("order_date")
   .parquet("lake/bronze_orders"))

Check the output: one directory per order_date=..., each holding a few .parquet files. That layout is the foundation Lab 1 builds on.

4. Common local-mode traps

Forgetting .master("local[*]") inside a script and hanging on cluster config.
collect() on a large DataFrame -> driver OOM. Use take(5) / show().
Default 200 shuffle partitions on a laptop: set spark.sql.shuffle.partitions to 4-8 for local runs.