Lab 1: Local PySpark Parquet ETL

Deliverable: a Git repo with an idempotent etl.py + partitioned Parquet lake + README.md with row counts and explain() output.

1. Business context

"ShopLite" dumps daily orders.csv (order_id, customer_id, order_date, status, amount). Build the bronze layer: clean, typed, partitioned Parquet that analysts can prune by date.

2. Requirements

1. etl.py runs with spark-submit etl.py --date 2026-09-20 in local mode.
2. Drops null-order_id rows; casts amount to decimal; normalizes status to lowercase.
3. Writes lake/bronze_orders/ partitioned by order_date (repartition to <= 8 files per date).
4. Re-runs are idempotent: same --date twice gives identical row counts (overwrite the date partition).
5. README.md includes: row counts in/out, rejected-row count, df.explain() excerpt, and file listing of one date partition.

3. Grading rubric (100 pts)

• Correctness & idempotency (30): re-run safe; counts reconcile.
• Partitioning & file sizing (25): date partitions, no tiny-file explosion.
• Code quality (20): lazy-plan discipline, no stray collect(), typed schema.
• README evidence (15): counts + plan excerpt + listing.
• Error handling (10): bad rows quarantined to lake/_quarantine/, not silently dropped.

4. Stretch

Add a --full-refresh flag that rewrites all partitions, and log per-partition counts to lake/_audit/etl_log.json.