Micro Free Course — 100% Free Learning
All lessons in this module are free to learn. Sign in with Google to save your progress.
Ingestion and Orchestration
Ingestion patterns (batch, CDC, streaming); managed tools (Fivetran, Airbyte, Meltano) and when to build vs buy; Apache Airflow deep dive; modern orchestrators (Dagster, Prefect); scheduling, backfills, and idempotency.
Module Content
Data Ingestion Patterns (Batch, CDC, Streaming)
Three ways data gets from source to warehouse. Each has different cost, complexity, and latency profile. FIND_VIDEO: search 'data ingestion patterns batch CDC streaming' — recommended channel: Confluent / Kahan Data Solutions. Aim for 10 min or under.
Recap — Picking the Right Ingestion Pattern
The decision tree for ingestion. Most pipelines use batch; CDC for relational sources; streaming reserved for narrow use cases.
Managed Ingestion Tools (Fivetran, Airbyte, Meltano)
Three tools, three philosophies. Pick by your team size, budget, and willingness to operate infrastructure. FIND_VIDEO: search 'fivetran vs airbyte vs meltano comparison' — recommended channel: Airbyte / Seattle Data Guy. Aim for 10 min or under.
Recap — When to Build vs Buy Ingestion
The build-vs-buy decision per source. Most pipelines mix managed connectors for SaaS and custom code for OLTP.
Apache Airflow — The Workhorse Orchestrator
The dominant orchestrator in 2026. Verbose, mature, omnipresent. Knowing it is non-negotiable for DE work. FIND_VIDEO: search 'apache airflow tutorial DAG basics' — recommended channel: Apache Airflow / Astronomer / Seattle Data Guy. Aim for 11 min or under.
Recap — Writing Maintainable Airflow DAGs
Airflow's primitives, the DAG concept, and the patterns that separate good DAGs from messy ones.
Modern Orchestrators (Dagster, Prefect)
Newer orchestrators address Airflow's pain points. Both are credible alternatives with different philosophies. FIND_VIDEO: search 'dagster vs prefect vs airflow comparison' — recommended channel: Dagster / Prefect / Kahan Data Solutions. Aim for 10 min or under.
Recap — When to Use Which Orchestrator
Three orchestrators, three philosophies. Pick by team preferences, team size, and what you're orchestrating.
Scheduling, Backfills, and Idempotency
Three concepts that separate amateur pipelines from production-grade ones. FIND_VIDEO: search 'data pipeline backfill idempotent design' — recommended channel: Seattle Data Guy / Kahan Data Solutions. Aim for 10 min or under.
Recap — Building Pipelines That Survive Re-runs
The triad: schedule, backfill, idempotency. Get them right and pipelines self-heal. Get them wrong and every retry is a crisis.