Recap — When to Use Which Orchestrator

# When to Use Which Orchestrator Airflow is the default. Dagster and Prefect are credible alternatives. Each has a different mental model and trade-offs. ## Airflow (review) Task-based DAGs. Workflows are explicit DAGs of tasks. The orchestrator schedules and runs tasks. Mental model: "I have a workflow with these steps and dependencies; run them in order." ## Dagster — asset-based orchestration Mental model: "I have data assets I produce; orchestration is about keeping them fresh and correct." The core abstraction is the **asset** — a data artifact (table, file, model) that depends on other assets. ```python from dagster import asset @asset def raw_orders(): return fetch_orders_from_source() @asset def cleaned_orders(raw_orders): return clean(raw_orders) @asset def daily_revenue(cleaned_orders): return aggregate_revenue(cleaned_orders) ``` The asset graph IS the pipeline. Dagster figures out dependencies automatically. Pros: - Asset-centric model matches data-team thinking. - Built-in observability for data lineage. - Strong testing story (Dagster has assertions and fixtures). - Type checking out of the box. - Modern Python (no DAG boilerplate). - Integrates with dbt natively (dbt models become Dagster assets). Cons: - Steeper learning curve coming from Airflow. - Smaller ecosystem (fewer pre-built integrations). - Less industry penetration; harder to hire for. When to use: - New projects without legacy Airflow. - Data engineering teams that think in assets/lineage. - Heavy dbt usage (Dagster + dbt is fantastic). - Modern Python practices appreciated. ## Prefect — task-based with Pythonic API Mental model: "Workflows are Python functions; orchestration runs them with retries, scheduling, observability." ```python from prefect import flow, task @task(retries=3, retry_delay_seconds=60) def extract(): return fetch_data() @task def transform(data): return clean(data) @task def load(data): insert_into_warehouse(data) @flow def daily_pipeline(): raw = extract() cleaned = transform(raw) load(cleaned) if __name__ == "__main__": daily_pipeline() ``` Pros: - Pythonic. Workflows are functions; tasks are functions. Very natural. - Lighter than Airflow for small projects. - Excellent retries, caching, and concurrency. - Cloud-native (Prefect Cloud is the SaaS). - Smaller infrastructure footprint. Cons: - Smaller ecosystem than Airflow. - Less of an industry standard. - Prefect Cloud cost can grow. When to use: - Small teams wanting clean Python without Airflow overhead. - Pipeline that's mostly Python code (not heavy SQL/Spark). - Teams that value developer experience. ## Decision matrix | Factor | Airflow | Dagster | Prefect | |---|---|---|---| | Industry penetration | High | Medium | Medium | | Asset-centric model | No | Yes | No | | dbt integration | OK | Native | OK | | Learning curve | Medium | Medium-high | Low | | Operational complexity | High | Medium | Low | | Type checking | Limited | Strong | Strong | | Hiring | Easy | Moderate | Moderate | | Cloud/managed offerings | MWAA, Composer, Astronomer | Dagster Cloud | Prefect Cloud | ## Which one for your team **Use Airflow when**: - You're at an enterprise (it's already standard). - You need the largest ecosystem of operators. - Your data engineers already know it (hiring matters). - You have ops capability to run it. **Use Dagster when**: - New project, no legacy. - Heavy dbt usage. - You want type-safe, asset-centric thinking. - Team is comfortable with newer tools. **Use Prefect when**: - Small team wanting minimal infrastructure. - Mostly Python pipelines. - You value developer experience. - You want to ship fast. ## Hybrid stacks Some teams use multiple: - Airflow for big batch jobs (dbt runs, Spark triggers). - Prefect for lightweight Python-heavy pipelines. - Or vice versa. Common when migrating: keep Airflow for legacy; new pipelines on Dagster/Prefect. ## What I'd choose in 2026 If starting greenfield with a small data team: **Prefect** for its Pythonic simplicity, or **Dagster** if the team is data-engineering-heavy and uses dbt heavily. If joining an existing team: whatever they have. Switching orchestrators is a years-long migration. ## Migration paths Airflow → Dagster: doable; dbt integration helps as a bridge. Airflow → Prefect: easier (similar task concept). Dagster ↔ Prefect: possible but unusual. In all cases, migration is months not weeks. Plan accordingly. ## Common orchestrator-selection mistakes 1. **Switching from Airflow without compelling reason.** Migration cost is high. 2. **Picking by hype.** Pick by team fit and use case. 3. **Self-hosting Airflow without ops capability.** Use MWAA or Astronomer. 4. **Underestimating Prefect/Dagster's value for small teams.** Airflow's overhead can be more than the entire pipeline. ## Takeaway Airflow = the standard. Dagster = asset-based, dbt-native, modern. Prefect = Pythonic, lightweight, dev-friendly. Pick by team