The Modern Data Engineering Landscape

Strip away the hype. Data engineering is moving data reliably from where it lives to where it can be used. FIND_VIDEO: search 'what is data engineering modern data stack' — recommended channel: Seattle Data Guy / Kahan Data Solutions. Aim for 10 min or under.

The 5-layer stack every modern data team operates. Knowing where you are on it shapes every decision.

The first design decision in any pipeline. Pick the wrong one and you build operational pain. FIND_VIDEO: search 'batch vs streaming data engineering when to use' — recommended channel: Confluent / Kahan Data Solutions. Aim for 10 min or under.

Three modes: batch, streaming, hybrid. Each has cost, complexity, and latency trade-offs. Most teams pick batch first and regret streaming.

The architectural shift that defines the modern data stack. Why pushing transformation INTO the warehouse changed everything. FIND_VIDEO: search 'ETL vs ELT difference modern data stack' — recommended channel: Kahan Data Solutions / Fivetran. Aim for 9 min or under.

ELT is more than a renaming. It changes how teams own data, deploy changes, and trust outputs.

The format wars and why they matter. Iceberg has emerged as the 2026 open lakehouse standard. FIND_VIDEO: search 'apache iceberg delta parquet lakehouse explained' — recommended channel: Tabular / Databricks / Kahan Data Solutions. Aim for 11 min or under.

The lakehouse is a real architecture. Parquet is the file; Iceberg/Delta is the table. Understanding the layers shapes 2026 builds.

Migrate a large raw S3 Parquet dataset into an Apache Iceberg lakehouse table. Implement hidden partitioning, demonstrate snapshot time travel, and configure small file compaction.