Layered Architecture — Source, Staging & Marts

This lesson on Layered Architecture — Source, Staging & Marts is hands-on and example-driven. You will design, implement, and maintain a multi-tier Bronze-Silver-Gold Lakehouse pipeline using Databricks Auto Loader, Delta Live Tables (DLT), and MLflow. You will establish an immutable raw audit layer, enforce schema and quality constraints at intermediate stages, and serve low-latency, business-ready aggregated outputs.

What You'll Be Able To Do

• Ingest raw, streaming, or batch files into an immutable Bronze Delta layer using Auto Loader.
• Declare Silver-layer cleaning and transformation pipelines using Delta Live Tables (DLT).
• Enforce declarative data quality constraints and quarantine bad records using DLT Expectations.
• Incorporate trained MLflow models as User-Defined Functions directly within Gold-layer DLT pipelines.
• Tune micro-batch throughput and latency using ingestion controls like maxFilesPerTrigger and watermarking.

Detailed Concept Walkthrough

1. Multi-Layered Architecture Foundations

The Bronze, Silver, Gold (BSG) layered architecture decouples raw ingestion from business transformation, providing a clear audit trail and modular pipeline debugging.

• Mechanism: Bronze serves as an immutable, append-only landing zone preserving raw source payload schemas; Silver cleans, standardizes, and models relational entities; Gold aggregates metrics and applies machine learning scoring for downstream consumers.
• Under the Hood: Data transitions through progressive validation states, moving from schema-on-read in Bronze to rigid, typed, and enriched schema enforcement in Silver and Gold.
• Best Practice: Maintain strict separation of concerns across engineering teams: Data Engineers manage Bronze ingestion pipelines, Analytics Engineers/Data Scientists maintain Silver feature transformations, and BI Analysts query Gold marts.

# Read raw incoming files incrementally into Bronze via Auto Loader
df_bronze = (spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/mnt/telemetry/_schema")
    .load("/mnt/raw_landing/"))

Key Takeaway: Layered architecture isolates ingestion failures from downstream business logic, guaranteeing complete pipeline reprocessability from immutable raw files.

2. Automated Bronze Ingestion with Auto Loader

Auto Loader incrementally and efficiently ingests new data files arriving in cloud storage without requiring manual state management or file listing.

• Mechanism: Configured via spark.readStream.format("cloudFiles"), Auto Loader automatically detects new files in object storage using either directory listing or cloud notification queues.
• Under the Hood: Auto Loader infers schemas on read and manages schema evolution by tracking schema changes in a configured checkpoint directory (cloudFiles.schemaLocation).
• Execution Flow: File processing scale is governed per micro-batch using parameters like maxFilesPerTrigger to prevent executor memory exhaustion during backfills.

import dlt

@dlt.table(
    name="bronze_orders",
    comment="Raw immutable orders stream"
)
def bronze_orders():
    return (
        spark.readStream
            .format("cloudFiles")
            .option("cloudFiles.format", "json")
            .option("cloudFiles.maxFilesPerTrigger", 1000)
            .load("/mnt/raw/orders/")
    )

Key Takeaway: Bronze pipelines must never filter or alter data during ingestion; they must act strictly as a preserved audit log.

3. Silver Transformation and DLT Expectations

Silver layers clean, deduplicate, and enforce data quality constraints using declarative Delta Live Tables (DLT) expectations.

• Mechanism: DLT allows developers to define DAGs of tables and views declaratively, automatically managing table creation, dependency resolution, and error handling.
• Syntax Rule: Apply @dlt.expect, @dlt.expect_or_drop, or @dlt.expect_or_fail decorators above DLT table definitions to enforce row-level validation constraints.
• Best Practice: Quarantine or fail invalid source records (such as negative prices or null customer IDs) at the Silver boundary before they propagate downstream to analytic consumers.

import dlt
from pyspark.sql.functions import col

@dlt.table(
    name="silver_orders_cleaned",
    comment="Standardized orders with validated constraints"
)
@dlt.expect_or_drop("valid_order_id", "order_id IS NOT NULL")
@dlt.expect_or_drop("positive_amount", "amount > 0")
def silver_orders_cleaned():
    return (
        dlt.read_stream("bronze_orders")
            .select(
                col("order_id").cast("long"),
                col("customer_id").cast("string"),
                col("amount").cast("double"),
                col("order_timestamp").cast("timestamp")
            )
    )

Key Takeaway: DLT Expectations embed programmatic data quality gates directly into pipelines without requiring custom monitoring infrastructure.

4. Gold Layer Aggregation and MLflow Integration

The Gold layer combines dimensional business aggregations with deployed machine learning models to deliver low-latency, actionable metrics.

• Mechanism: Gold tables summarize Silver data by business keys (e.g., daily revenue, customer lifetime value) and enrich records with predictive inference.
• Under the Hood: Trained models logged in the MLflow Model Registry are loaded as Spark User-Defined Functions (UDFs) and evaluated directly within distributed batch or streaming queries.
• Execution Flow: Streaming Gold layers manage continuous state and late-arriving event processing using time-based watermarking to balance latency and computation cost.

import dlt
import mlflow.pyfunc
from pyspark.sql.functions import struct

# Load logged MLflow model as a Spark UDF
fraud_model_udf = mlflow.pyfunc.spark_udf(spark, "models:/FraudDetection/Production")

@dlt.table(
    name="gold_flagged_transactions",
    comment="Scored transactions for fraud operations"
)
def gold_flagged_transactions():
    silver_df = dlt.read("silver_orders_cleaned")
    return silver_df.withColumn(
        "fraud_prediction", 
        fraud_model_udf(struct(*[col(c) for c in silver_df.columns]))
    )

Key Takeaway: Gold layer tables deliver high-value, model-enriched, and aggregated data tailored directly for analytical tools and operational systems.

Topics Covered in Layered Architecture — Source, Staging & Marts

1. Architectural Overview and Benefits (0:00 - 1:02) — Covers the fundamental principles, scalability, and CI/CD benefits of implementing the Lakehouse multi-layer pattern.
2. Bronze, Silver, and Gold Definitions (1:02 - 2:56) — Details the state of data and purpose across raw, refined, and business-ready layers.
3. Role Mapping and Ownership (2:56 - 4:36) — Assigns layer responsibilities across data engineers, analytics engineers, data scientists, and BI analysts.
4. Tooling and Pipeline Construction (4:36 - 6:39) — Demonstrates Auto Loader ingestion, Delta Live Tables declarations, and MLflow UDF integration.
5. Data Quality and DLT Expectations (6:39 - 8:12) — Shows how to enforce declarative constraints and manage schema validation across layers.
6. Latency and Throughput Tuning (8:13 - 9:29) — Explains techniques for controlling micro-batch processing speed, trigger rates, and streaming watermarks.

Data Modeling & Warehousing Fundamentals Cheat Sheet

• spark.readStream.format("cloudFiles") — Configures Auto Loader for streaming cloud file ingestion

  spark.readStream.format("cloudFiles").option("cloudFiles.format", "json").load("/path")
  

• cloudFiles.maxFilesPerTrigger — Limits files read per micro-batch trigger

  .option("cloudFiles.maxFilesPerTrigger", 500)
  

• @dlt.table — Declares a Delta Live Table definition

  @dlt.table(name="silver_table", comment="Cleaned data")
  

• @dlt.expect_or_drop — Enforces constraint; drops violating rows from downstream

  @dlt.expect_or_drop("valid_id", "id IS NOT NULL")
  

• @dlt.expect_or_fail — Enforces constraint; halts pipeline execution on violation

  @dlt.expect_or_fail("valid_total", "total >= 0")
  

• dlt.read_stream() — Reads upstream DLT table as continuous stream

  df = dlt.read_stream("bronze_events")
  

• mlflow.pyfunc.spark_udf() — Registers MLflow model as distributed Spark UDF

  predict_udf = mlflow.pyfunc.spark_udf(spark, "models:/MyModel/1")
  

Comparison Table

Common Pitfalls

• Mistake: Applying transformations, deduplication, or filtering during Bronze ingestion. Avoid: Ingest raw data as-is into Bronze to maintain an immutable audit trail.
• Mistake: Overwriting Silver tables rather than streaming incremental changes. Avoid: Use declarative streaming reads and incremental merges to preserve pipeline lineage and efficiency.
• Mistake: Applying heavy ML model scoring on uncleaned Bronze tables. Avoid: Invoke MLflow Spark UDFs strictly within Silver or Gold layers after enforcing schema validation.

FAQs

• Why should Bronze tables remain completely untransformed? Preserving raw source files unchanged allows you to recreate downstream Silver and Gold tables from scratch if downstream transformation logic or schemas change.
• How do DLT Expectations handle invalid records? Depending on the decorator used, DLT can log the violation while keeping the record (@dlt.expect), drop the invalid record (@dlt.expect_or_drop), or abort the pipeline execution (@dlt.expect_or_fail).
• How do you control ingestion backpressure when processing large file backlogs? Set the cloudFiles.maxFilesPerTrigger option in your Auto Loader configuration to cap the number of files ingested per micro-batch.