Airflow in Production — Scheduling, Backfills & Retries

Three concepts that separate amateur pipelines from production-grade ones. FIND_VIDEO: search 'data pipeline backfill idempotent design' — recommended channel: Seattle Data Guy / Kahan Data Solutions. Aim for 10 min or under.

23 minutesVideo LessonPDF notes

Watch Airflow in Production — Scheduling, Backfills & Retries on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Backfilling Core Concepts — Introduces backfilling as the process of running DAGs or tasks for past, untriggered, or failed intervals.
Backfill Scenarios and Use Cases — Outlines key scenarios requiring backfills including new pipelines, failure handling, logic changes, and selective reruns.
CLI Backfill Tooling — Explains that backfilling is officially executed via the Airflow CLI rather than through the web interface.
Scheduler Catchup Mechanism — Demonstrates enabling catchup=True on a DAG to auto-generate historical runs between start_date and current time.
Date-Bounded CLI Backfills — Executes a CLI backfill using start and end date flags and verifies the Backfill run type indicator in the UI.
Diagnosing Failed Intervals — Identifies failed historical date windows using the Dag Runs table and the Calendar View in the web UI.
Task Retries and Try Numbers — Explains Try Number tracking and shows how standard backfills fail to rerun previously failed tasks.
Rerunning Failed Tasks — Applies the rerun-failed-tasks flag to force a single rerun attempt on failed tasks within the date window.
Resetting DAG Run States — Combines reset-dag-runs with rerun-failed-tasks to reset execution state and enforce full configured task retries.
Disabling Automatic Retries — Introduces the disable-retry flag to bypass task retry delays and optimize backfill execution speed.

PDF notes

Bottom line: You will be able to execute targeted historical DAG backfills from the Airflow CLI, diagnose failed execution windows via the web UI, and manage task retry behaviors using state-resetting and rerun flags.

This lesson on Airflow in Production — Scheduling, Backfills & Retries is hands-on and example-driven. You will be able to execute targeted historical DAG backfills from the Airflow CLI, diagnose failed execution windows via the web UI, and manage task retry behaviors using state-resetting and rerun flags.

What You'll Be Able To Do

Configure catchup parameters on DAG definitions to control automatic historical run generation upon unpausing.
Execute CLI backfills over targeted date ranges using the airflow dags backfill command with start and end flags.
Diagnose pipeline execution failure intervals using the Airflow UI Dag Runs list and Calendar View.
Rerun failed task instances within historical execution windows using the --rerun-failed-tasks flag.
Reset DAG Run states with --reset-dag-runs to enforce configured task retry limits during CLI backfills.
Accelerate historical reprocessing by bypassing task retries using the --disable-retry flag.

Detailed Concept Walkthrough

1. Automated Catchup vs Manual Backfills

Historical execution intervals are generated automatically by scheduler catchup or manually targeted via CLI backfill commands.

Mechanism: Setting catchup=True on a DAG configuration instructs the scheduler to create runs for every missing interval between start_date and current time upon unpausing.
Execution Flow: When processing intervals prior to start_date or applying selective reprocessing, manual CLI backfilling bypasses the automatic scheduler queue.
Best Practice: Keep catchup=False as the default in production DAG definitions to prevent accidental execution floods when deploying new pipelines.

from datetime import datetime, timedelta
from airflow import DAG

with DAG(
    dag_id="sales_ingest_pipeline",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
    catchup=False,  # Disables automatic historical backlog execution
    default_args={"retries": 3, "retry_delay": timedelta(minutes=5)},
) as dag:
    pass

Key Takeaway: Automatic catchup runs intervals from start_date upon unpausing, whereas CLI backfilling manually targets custom time windows.

2. Executing CLI Historical Backfills

The Airflow CLI is the required interface for triggering custom backfills across specified historical date bounds.

Syntax Rule: The airflow dags backfill command requires a DAG ID along with -s (--start-date) and -e (--end-date) flags formatted as YYYY-MM-DD.
Under the Hood: CLI backfills directly create execution jobs tagged with a distinct 'Backfill' run type visible in the Airflow UI.
Best Practice: Check the Dag Runs list or Calendar View in the web UI first to determine the exact failure dates before formulating CLI parameters.

# Backfill a specific historical date range prior to or within DAG start_date
airflow dags backfill \
    --start-date 2025-01-25 \
    --end-date 2025-02-05 \
    sales_ingest_pipeline

Key Takeaway: Backfills cannot be initiated directly through the web UI and must be run using the Airflow CLI.

3. Task Retries and Failed State Handling

Standard backfill invocations skip already failed runs unless rerun flags are explicitly supplied.

Mechanism: Every task execution increments its Try Number metadata field until reaching the maximum configured retries parameter.
Under the Hood: Running a standard airflow dags backfill against previously executed intervals ignores failed tasks without incrementing the Try Number.
Nuance: Adding --rerun-failed-tasks forces Airflow to rerun failed tasks, but it only attempts a single execution per failed run attempt.

# Re-execute only the failed tasks across a historical interval (single retry attempt)
airflow dags backfill \
    --start-date 2025-01-25 \
    --end-date 2025-02-05 \
    --rerun-failed-tasks \
    sales_ingest_pipeline

Key Takeaway: A standard backfill ignores existing failed tasks unless the --rerun-failed-tasks flag is specified.

4. Resetting DAG Run States and Disabling Retries

Resetting run states restores full task retry counts, whereas disabling retries optimizes backfills by failing immediately.

Mechanism: The --reset-dag-runs flag clears existing DAG run state in the metadata database, enabling tasks to use their full retries allowance.
Execution Flow: Combining --rerun-failed-tasks with --reset-dag-runs guarantees that tasks with transient errors retry up to the configured retry limit.
Best Practice: Use --disable-retry during high-volume backfills to fail fast on deterministic bugs and prevent long retry delays.

# Option A: Full recovery respecting configured retry counts (e.g., retries=15)
airflow dags backfill -s 2025-01-25 -e 2025-02-05 --rerun-failed-tasks --reset-dag-runs sales_ingest_pipeline

# Option B: Fast-fail optimization bypassing automatic task retries
airflow dags backfill -s 2025-01-25 -e 2025-02-05 --rerun-failed-tasks --disable-retry sales_ingest_pipeline

Key Takeaway: Resetting DAG runs restores full task retry limits, while disabling retries avoids bottlenecks during backfills.

Topics Covered in Airflow in Production — Scheduling, Backfills & Retries

Backfilling Core Concepts (0:00 - 0:33) — Introduces backfilling as the process of running DAGs or tasks for past, untriggered, or failed intervals.
Backfill Scenarios and Use Cases (0:33 - 1:15) — Outlines key scenarios requiring backfills including new pipelines, failure handling, logic changes, and selective reruns.
CLI Backfill Tooling (1:15 - 1:42) — Explains that backfilling is officially executed via the Airflow CLI rather than through the web interface.
Scheduler Catchup Mechanism (1:42 - 2:24) — Demonstrates enabling catchup=True on a DAG to auto-generate historical runs between start_date and current time.
Date-Bounded CLI Backfills (2:24 - 4:11) — Executes a CLI backfill using start and end date flags and verifies the Backfill run type indicator in the UI.
Diagnosing Failed Intervals (4:11 - 5:24) — Identifies failed historical date windows using the Dag Runs table and the Calendar View in the web UI.
Task Retries and Try Numbers (5:24 - 7:11) — Explains Try Number tracking and shows how standard backfills fail to rerun previously failed tasks.
Rerunning Failed Tasks (7:11 - 7:54) — Applies the rerun-failed-tasks flag to force a single rerun attempt on failed tasks within the date window.
Resetting DAG Run States (7:54 - 8:52) — Combines reset-dag-runs with rerun-failed-tasks to reset execution state and enforce full configured task retries.
Disabling Automatic Retries (8:52 - 9:12) — Introduces the disable-retry flag to bypass task retry delays and optimize backfill execution speed.

Data Engineering Cheat Sheet

catchup=True — Auto-generates past runs between start_date and unpause time
```
DAG(dag_id="orders", start_date=datetime(2025, 1, 1), catchup=True)
```
airflow dags backfill -s <START> -e <END> <DAG_ID> — Triggers historical execution for designated date range
```
airflow dags backfill -s 2025-01-01 -e 2025-01-07 daily_etl
```

--rerun-failed-tasks — Attempts a single rerun of failed tasks in backfill window

airflow dags backfill -s 2025-01-01 -e 2025-01-07 --rerun-failed-tasks daily_etl

--reset-dag-runs — Resets run metadata allowing full task retry counts

airflow dags backfill -s 2025-01-01 -e 2025-01-07 --rerun-failed-tasks --reset-dag-runs daily_etl

--disable-retry — Disables task retries for faster failing backfill execution
```
airflow dags backfill -s 2025-01-01 -e 2025-01-07 --disable-retry daily_etl
```

Comparison Table

Backfill Option	Retry Policy Behavior	Primary Use Case
Default Backfill	Skips existing runs completely	Generating new untriggered intervals
--rerun-failed-tasks	Executes failed tasks once	Single-attempt rerun of failures
--reset-dag-runs	Enforces full task retries	Complete rerun with full retry policy
--disable-retry	Overrides task retries to zero	Fast-failing backfills during debugging

Common Pitfalls

Mistake: Attempting to trigger historical backfills directly through the Airflow web UI. Avoid: Execute the airflow dags backfill command via the CLI using date flags.
Mistake: Running a basic backfill expecting failed tasks to rerun automatically. Avoid: Provide the rerun-failed-tasks flag to target tasks in a failed state.
Mistake: Expecting rerun-failed-tasks alone to respect configured retry limits. Avoid: Include reset-dag-runs alongside rerun-failed-tasks to reset state and restore retry counts.
Mistake: Allowing long retry delays during large historical backfills with broken logic. Avoid: Pass disable-retry to terminate failing tasks immediately without retry waits.

FAQs

Can I trigger a backfill operation directly from the Airflow web UI? No, official Airflow backfill operations must be executed exclusively through the Command Line Interface (CLI).
What is the difference between catchup=True and a CLI backfill? Setting catchup=True prompts the scheduler to run all missing intervals since start_date upon unpausing, whereas CLI backfilling manually runs any user-defined date window.
Why does --rerun-failed-tasks only retry my failing task once despite retries=15? The --rerun-failed-tasks flag only triggers a single attempt; you must also pass --reset-dag-runs to restore the DAG run state and full retry policy.
How can I locate the exact date bounds of failed runs to backfill? Inspect the Dag Runs list or use the Calendar View in the Airflow UI to find the start and end dates of failed executions.

Common pitfalls

Mistake: Attempting to trigger historical backfills directly through the Airflow web UI. Avoid: Execute the airflow dags backfill command via the CLI using date flags.
Mistake: Running a basic backfill expecting failed tasks to rerun automatically. Avoid: Provide the rerun-failed-tasks flag to target tasks in a failed state.
Mistake: Expecting rerun-failed-tasks alone to respect configured retry limits. Avoid: Include reset-dag-runs alongside rerun-failed-tasks to reset state and restore retry counts.
Mistake: Allowing long retry delays during large historical backfills with broken logic. Avoid: Pass disable-retry to terminate failing tasks immediately without retry waits.

This tutorial provides a practical, step-by-step guide to using **Backfilling Mechanisms** within *Apache Airflow*. After watching, a learner will be able to manage historical execution, rerun failed pipeline components, and optimize the retry behavior of their Directed Acyclic Graphs (DAGs). ### **Curriculum Alignment: Topics Covered** 1. **Introduction to Backfilling Concepts (00:00 - 00:33)** * **Concept:** Backfilling is the process of manually running a DAG or specific tasks for past, untriggered, or failed execution intervals. * **Action:** Defining the core mechanism and necessity of backfilling (e.g., adding a new task to an existing pipeline). 2. **Key Use Cases and Optimization Strategies (00:33 - 01:15)** * **Concept:** Identifies four primary scenarios requiring backfilling: new DAG creation, handling failures, applying logic modifications (e.g., new tasks), and optimizing reruns by targeting specific subsets of tasks. * **Action:** Listing scenarios where backfilling is the appropriate operational procedure. 3. **Airflow Backfilling Tooling (CLI) (01:15 - 01:42)** * **Concept:** The official method for initiating a backfill operation in Airflow is exclusively through the Command Line Interface (CLI), specifically the `airflow dags backfill` command, not the UI. * **Action:** Introducing the `airflow dags backfill` command and stressing the importance of reviewing its documentation for various options. 4. **Generating Historical Runs (The Catchup Mechanism) (01:42 - 02:24)** * **Concept:** Demonstrating the prerequisite step of setting `catchup=True` to automatically generate historical DAG runs between the `start_date` and the moment the DAG is unpaused. * **Action:** Modifying a DAG file to set `catchup=True` and observing the resulting historical runs in the UI. 5. **Practical Application: Backfilling a New Pipeline (CLI Execution) (02:24 - 04:11)** * **Concept:** Executing a backfill command to run a DAG for an interval *prior* to its defined start date using the `-s` (start) and `-e` (end) flags. * **Action:** Running `airflow dags backfill -s YYYY-MM-DD -e YYYY-MM-DD <dag_id>`, and verifying the resulting "Backfill" run type indicator in the Airflow UI. 6. **Diagnosing Failed Intervals for Rerun (04:11 - 05:24)** * **Concept:** Establishing a failure scenario (raising an exception) and demonstrating two methods to find the exact failed date range: navigating the Dag Runs list or using the Calendar View. * **Action:** Diagnosing the failure start and end dates (e.g., Jan 25 to Feb 5) in the Airflow UI to define the required backfill window. 7. **Understanding Task Retries and the Try Number (05:24 - 07:11)** * **Concept:** Explaining the *Try Number* (which increases with every execution attempt) and the *Retries* parameter (the maximum allowed attempts before a task is marked failed). Critically, demonstrating that a simple backfill execution does not automatically rerun failed tasks. * **Action:** Observing the Try Number in the UI and running a basic backfill command that fails to change the try count. 8. **Implementing Limited Reruns (`--rerun-failed-tasks`) (07:11 - 07:54)** * **Concept:** Introducing the `--rerun-failed-tasks` option, which forces the backfill process to attempt to rerun only the tasks marked as failed. Notes that this option only executes the task once per failed DAG run attempt. * **Action:** Executing backfill with the `--rerun-failed-tasks` flag and observing a single increment in the task's Try Number. 9. **Respecting Full Retry Limits (`--reset-dag-runs`) (07:54 - 08:52)** * **Concept:** Explaining that to fully utilize the DAG's defined retry policy (e.g., retrying 15 times), the DAG Run state itself must be reset, requiring the `--reset-dag-runs` flag. * **Action:** Executing backfill with both `--rerun-failed-tasks` and `--reset-dag-runs` to ensure Airflow respects the defined task retries parameter. 10. **Optimization: Disabling Automatic Retries (08:52 - 09:12)** * **Concept:** Recognizing that respecting the full retry count can dramatically slow down the backfilling process, and offering the `--disable-retry` flag as an optimization alternative. * **Action:** Introducing the `--disable-retry` option for backfills where automatic retries are not desired. ### **Learning Outcomes** * **Proficiency:** Define and execute CLI-based backfilling operations for specific date ranges and DAG IDs, using flags like `-s` and `-e`. * **Workflow Management:** Successfully diagnose and recover failed data pipelines by correctly identifying failure intervals and applying state-resetting parameters (`--reset-dag-runs`). * **Tooling:** Comfortably navigate the Airflow UI (Dag Runs, Calendar View) for diagnosing errors and verifying the status and run type of backfilled executions. ### **Advanced/Tricky Concepts** * **Backfilling vs. Retries:** By default, the `airflow dags backfill` command will NOT respect the `retries` parameter set on tasks unless the user explicitly adds the `--reset-dag-runs` flag to reset the entire DAG Run state. Simply using `--rerun-failed-tasks` only attempts a single retry.

Frequently asked questions

Can I trigger a backfill operation directly from the Airflow web UI?

No, official Airflow backfill operations must be executed exclusively through the Command Line Interface (CLI).

What is the difference between catchup=True and a CLI backfill?

Setting catchup=True prompts the scheduler to run all missing intervals since start_date upon unpausing, whereas CLI backfilling manually runs any user-defined date window.

Why does --rerun-failed-tasks only retry my failing task once despite retries=15?

The --rerun-failed-tasks flag only triggers a single attempt; you must also pass --reset-dag-runs to restore the DAG run state and full retry policy.

How can I locate the exact date bounds of failed runs to backfill?

Inspect the Dag Runs list or use the Calendar View in the Airflow UI to find the start and end dates of failed executions.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.