Micro Free Course — 100% Free Learning
All lessons in this module are free to learn. Sign in with Google to save your progress.
Reliability and Production Patterns
Data quality testing (Great Expectations, dbt tests); pipeline observability (Monte Carlo, OpenLineage, Datadog); streaming basics (Kafka, Pub/Sub, Kinesis); Spark for batch and structured streaming.
Module Content
Data Quality Testing (Great Expectations, dbt Tests)
Two dominant approaches to encoding data quality rules. Most production stacks use both. FIND_VIDEO: search 'great expectations vs dbt tests data quality' — recommended channel: Great Expectations / dbt Labs. Aim for 10 min or under.
Recap — Building a Data Quality Discipline
Data quality is the bridge between 'pipeline ran' and 'data is correct'. Encode rules, run them every load, alert on failures.
Pipeline Observability (OpenLineage, Monte Carlo, Datafold)
Beyond pass/fail tests: end-to-end visibility into pipeline health, lineage, and data freshness. FIND_VIDEO: search 'data observability openlineage monte carlo pipeline' — recommended channel: Monte Carlo / OpenLineage. Aim for 10 min or under.
Recap — Observability for Data Pipelines
What 'observability' means in DE: lineage, freshness, volume, schema drift, distribution drift. Tools that surface them.
Streaming Basics (Kafka, Pub/Sub, Kinesis)
The streaming layer of the modern stack. Covers Kafka concepts, when to use streaming, and operational realities. FIND_VIDEO: search 'kafka streaming basics tutorial data engineering' — recommended channel: Confluent / Stephane Maarek. Aim for 10 min or under.
Recap — Streaming Without Drowning
Streaming concepts (topics, partitions, consumer groups, offsets) and when streaming is justified vs batch.
Spark for Data Engineers
When and how to use Spark for batch transformations. The default for large-data ELT outside the warehouse. FIND_VIDEO: search 'apache spark tutorial data engineering' — recommended channel: Apache Spark / Databricks. Aim for 11 min or under.
Recap — When Spark Belongs in Your Stack
Spark for DE: when warehouse SQL isn't enough; PySpark patterns; Databricks vs open-source Spark; when not to use Spark.