Micro Free Course — 100% Free Learning

All lessons in this module are free to learn. Sign in with Google to save your progress.

Ingestion and Orchestration

Ingestion patterns (batch, CDC, streaming); managed tools (Fivetran, Airbyte, Meltano) and when to build vs buy; Apache Airflow deep dive; modern orchestrators (Dagster, Prefect); scheduling, backfills, and idempotency.

Module Progress0% Complete

110 min total

5 Lessons

0 Completed

Module Content

Change Data Capture (CDC) — Real-Time Database Replication

Three ways data gets from source to warehouse. Each has different cost, complexity, and latency profile. FIND_VIDEO: search 'data ingestion patterns batch CDC streaming' — recommended channel: Confluent / Kahan Data Solutions. Aim for 10 min or under.

9 minVideo

Start

Quiz: Ingestion Mechanics — Change Data Capture

The decision tree for ingestion. Most pipelines use batch; CDC for relational sources; streaming reserved for narrow use cases.

10 minTutorial

Start

Case 2 — Real-Time Ingestion: Postgres CDC to Cloud Warehouse

Deploy a production-grade Change Data Capture (CDC) pipeline streaming OLTP database updates to an analytical warehouse using Debezium, Kafka, and replication slots.

90 minSubmission

Start

Managed Ingestion Tools (Fivetran, Airbyte, Meltano)

Three tools, three philosophies. Pick by your team size, budget, and willingness to operate infrastructure. FIND_VIDEO: search 'fivetran vs airbyte vs meltano comparison' — recommended channel: Airbyte / Seattle Data Guy. Aim for 10 min or under.

12 minVideo

Start

Quiz: Build vs Buy Ingestion

The build-vs-buy decision per source. Most pipelines mix managed connectors for SaaS and custom code for OLTP.