Local PySpark Setup & First DataFrame Job

# Local PySpark Setup & Your First DataFrame Job You do not need a cluster for this course. A laptop with Python 3.10+ and 8 GB RAM runs every lab. We use local-mode Spark (`master("local[*]")`). ## 1. Install (5 minutes) ```bash pip install "pyspark==3.5.*" pandas pyarrow delta-spark java -version # needs Java 8/11/17 on PATH; Spark fails without it ``` Verify: ```python from pyspark.sql import SparkSession spark = (SparkSession.builder.appName("smoke") .master("local[*]") .config("spark.sql.shuffle.partitions", "4") # small local shuffle .getOrCreate()) print(spark.version) ``` ## 2. The 4 DataFrame rules 1. **Transformations are lazy** — `filter/select/join` build a plan; nothing runs until an **action** (`show/count/write/parquet`) triggers it. 2. **Read the plan, not the code** — `df.explain()` shows whether Spark will scan, shuffle, or broadcast. A lab that cannot explain its plan is incomplete. 3. **Narrow vs wide transforms** — `filter/select/withColumn` need no shuffle; `groupBy/join/distinct` shuffle data across partitions and dominate runtime. 4. **Partitions are parallelism** — each partition is one task. Too few (1-2) = idle cores; thousands of tiny files = driver overhead. Repartition before writes. ## 3. First job: CSV -> partitioned Parquet ```python df = spark.read.option("header", True).csv("data/orders.csv") (df.filter("status = 'completed'") .repartition(4, "order_date") .write.mode("overwrite") .partitionBy("order_date") .parquet("lake/bronze_orders")) ``` Check the output: one directory per `order_date=...`, each holding a few `.parquet` files. That layout is the foundation Lab 1 builds on. ## 4. Common local-mode traps - Forgetting `.master("local[*]")` inside a script and hanging on cluster config. - `collect()` on a large DataFrame -> driver OOM. Use `take(5)` / `show()`. - Default 200 shuffle partitions on a laptop: set `spark.sql.shuffle.partitions` to 4-8 for local runs.