Recap — When Spark Belongs in Your Stack

# When Spark Belongs in Your Stack Apache Spark is the dominant distributed computing framework for batch data processing. The question is when to reach for it. ## When you DO need Spark 1. **Data larger than the warehouse can handle.** Warehouse-side processing of multi-TB tables can be slow or expensive. 2. **File-based processing.** Source data is in S3/GCS as Parquet/JSON; Spark processes directly without loading to warehouse. 3. **Complex transformations.** Custom Python logic per row at scale; warehouse SQL gets unwieldy. 4. **ML pipelines.** Feature engineering at scale; integrates with MLflow. 5. **Cost optimization.** Sometimes Spark on spot instances is cheaper than equivalent warehouse compute. ## When you DON'T need Spark 1. **Data fits in warehouse comfortably.** Snowflake/BigQuery/Redshift do the same transformation in fewer minutes with no infra. 2. **dbt covers your transformations.** SQL is faster to write/maintain than PySpark. 3. **Small team without Spark expertise.** Operational tax is significant. 4. **Pipelines are simple ELT.** Loading raw + dbt is enough. The 2026 reality: cloud warehouses are powerful enough that most pipelines don't need Spark. Spark earns its place when warehouse processing is too slow, too expensive, or insufficient (need custom Python). ## Spark concepts ### Cluster Driver + workers. Driver coordinates; workers execute. You write a Spark "application"; it runs on the cluster. ### DataFrame Distributed table abstraction. Like pandas but partitioned across workers. ```python from pyspark.sql import SparkSession spark = SparkSession.builder.appName("MyJob").getOrCreate() df = spark.read.parquet("s3://bucket/data/") result = ( df.filter("status = 'paid'") .groupBy("country") .agg({"amount": "sum"}) .orderBy("country") ) result.write.parquet("s3://bucket/output/") ``` ### Transformations vs actions - **Transformations** (filter, groupBy, join) are lazy. They build a plan. - **Actions** (count, write, collect) execute the plan. Spark optimizes the plan before executing. ### Partitions DataFrame data is split into partitions. Each partition processed by one task. Right number of partitions = balance between parallelism and overhead. Typically 100-1000s for large jobs. ### Shuffle Operations that redistribute data across workers (groupBy, join). Expensive. Avoid when possible. ```python # Shuffle-heavy: group across all data df.groupBy("country").sum("amount") # Less shuffle: filter first to reduce data df.filter("amount > 1000").groupBy("country").sum("amount") ``` ## PySpark patterns ### Read formatted data ```python df_parquet = spark.read.parquet("s3://path/") df_json = spark.read.json("s3://path/") df_csv = spark.read.csv("s3://path/", header=True) df_iceberg = spark.read.format("iceberg").load("table_name") df_jdbc = spark.read.jdbc(url, table) ``` ### Common transformations ```python from pyspark.sql import functions as F df.filter(F.col("status") == "paid") df.withColumn("amount_usd", F.col("amount") * 0.012) (df.groupBy("country") .agg(F.sum("amount").alias("total"), F.count("*").alias("count"))) df_orders.join(df_customers, on="customer_id", how="inner") from pyspark.sql.window import Window w = Window.partitionBy("customer_id").orderBy("order_date") df.withColumn("rank", F.row_number().over(w)) ``` ### Write outputs ```python df.write .mode("overwrite") .partitionBy("year", "month") .parquet("s3://output/") ``` ## Spark deployment options ### Databricks Managed Spark on cloud. Notebook-friendly. Industry standard for enterprise. Pros: managed, integrated with MLflow/Unity Catalog, performant. Cons: expensive, vendor-locked. ### AWS EMR / GCP Dataproc / Azure HDInsight Managed Spark from cloud vendors. Cheaper than Databricks; less polished. Pros: managed-ish, integrates with cloud-native services. Cons: more ops than Databricks. ### Self-hosted Spark (Kubernetes) Open-source Spark on K8s. Pros: cheapest, full control. Cons: ops burden significant. ### Serverless Spark - AWS Glue (serverless Spark for ETL). - GCP Dataproc Serverless. - Databricks Serverless. Pros: no cluster management; pay per use. Cons: can be expensive for steady workloads; cold-start latency. For most teams in 2026: Databricks or serverless Spark for batch jobs. ## Spark vs warehouse vs Python Decision framework: ``` Data size + transformation complexity: ├── 10TB, SQL → Spark or warehouse (depending on cost) └── > 10TB, complex → Spark (no alternative) ``` For most production pipelines: warehouse + dbt covers it. Spark for the exceptions. ## Modern alternatives to Spark - **Polars** — single-node, Rust-based DataFrame. Surprisingly fast on machines with 100GB+ RAM. Replaces Spark for "medium" data. - **DuckDB** —