Apache Airflow Explained: How to Orchestrate Data Pipelines

Apache Airflow is an open-source platform for data pipeline orchestration: you define workflows as Python code, and Airflow schedules them, runs each step in the right order, retries failures, and shows you what happened. This Apache Airflow tutorial covers the core ideas you need to write reliable pipelines, including DAGs, tasks, operators, the scheduler, retries, backfills, and why every task should be idempotent. Airflow does not move or transform data itself; it coordinates the systems that do.

What is data pipeline orchestration?

A real data pipeline is rarely one script. It is a sequence of steps: extract from an API, land raw files in object storage, load them into a warehouse, run transformations, test the results, and refresh a dashboard. Each step depends on earlier ones, each can fail, and the whole thing must run on a schedule.

Orchestration is the layer that manages those dependencies. An orchestrator answers questions like: what runs next, what happens if a step fails, which runs are late, and how do I rerun last Tuesday without breaking today? Cron can start a script on a schedule, but it knows nothing about dependencies, retries, history, or reprocessing past dates. That gap is what Airflow fills.

Core Airflow concepts

DAGs

A DAG (directed acyclic graph) is a workflow. Nodes are tasks and edges are dependencies. "Acyclic" means there are no loops, so there is always a valid order in which to run the tasks. A DAG also carries scheduling information: when it starts, how often it runs, and how to handle missed intervals.

Tasks and operators

A task is one unit of work inside a DAG. An operator is a template for a kind of task. Airflow ships with general operators such as BashOperator and PythonOperator, and provider packages add operators and hooks for cloud services, databases, and tools such as Spark, dbt, and Kubernetes. Sensors are a special kind of operator that wait for a condition, like a file appearing in a bucket.

Scheduler, executor, and workers

The scheduler parses DAG files, decides which task instances are ready to run, and hands them to an executor. The executor determines where tasks run: locally, on a Celery worker pool, or as individual Kubernetes pods. The metadata database stores the state of every DAG run and task instance, and the web UI reads from it so you can see history, logs, and failures.

Data intervals and logical dates

Each scheduled DAG run covers a data interval. A daily DAG run for a given logical date processes data for that day and typically starts after the interval ends. This is the source of a common beginner surprise: the run labeled with yesterday's date executes today, because yesterday's data is only complete once yesterday is over. Use the interval boundaries that Airflow provides, such as data_interval_start and data_interval_end, instead of calling now() in your task code.

Writing a DAG with the TaskFlow API

The TaskFlow API lets you write tasks as decorated Python functions. Return values are passed between tasks through XCom automatically, and dependencies are inferred from function calls. The example below uses the Airflow 2.x import path; in Airflow 3 the same decorators are imported from airflow.sdk.

from datetime import timedelta

import pendulum
from airflow.decorators import dag, task


@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    max_active_runs=1,
    default_args={"retries": 3, "retry_delay": timedelta(minutes=5)},
    tags=["orders"],
)
def orders_daily():

    @task
    def extract(data_interval_start=None, data_interval_end=None) -> str:
        # Pull only the rows for this run's interval and land them at a
        # deterministic path, so a rerun overwrites instead of duplicating.
        day = data_interval_start.strftime("%Y-%m-%d")
        path = f"s3://example-bucket/raw/orders/dt={day}/orders.parquet"
        # ... call the source API for [data_interval_start, data_interval_end) ...
        return path

    @task
    def load(path: str) -> str:
        # ... replace the matching partition in the warehouse table ...
        return path

    @task
    def validate(path: str) -> None:
        # ... check row counts and null rates; raise to fail the task ...
        pass

    validate(load(extract()))


orders_daily()

A few details matter here. Context values like data_interval_start are injected when you declare them as parameters. XCom is meant for small values such as file paths or IDs, not datasets; pass references to data, not the data itself. And max_active_runs=1 keeps runs from overlapping when a single run writes to shared tables.

How do retries work in Airflow?

When a task raises an exception, Airflow marks it as failed or schedules a retry if retries remain. You can set retries, retry_delay, and retry_exponential_backoff per task or through default_args. Retries are ideal for transient failures like network timeouts or rate limits. They do nothing for bad logic, so combine them with alerting through failure callbacks or notifications. The same ideas are covered in more depth in retry mechanisms.

Retries have one important prerequisite: running a task twice must be safe. That leads to the most important rule in Airflow.

Why Airflow tasks must be idempotent

An idempotent task produces the same result whether it runs once or five times for the same interval. Airflow will rerun tasks through retries, manual clears, and backfills, so non-idempotent tasks eventually create duplicates or corrupt data.

Practical ways to make tasks idempotent:

• Scope each run to its data interval instead of "everything since the last run".
• Write to deterministic locations, such as a partition keyed by date, and overwrite that partition.
• Use MERGE or delete-then-insert inside a transaction rather than blind INSERT.
• Avoid side effects that cannot be repeated, or guard them with an idempotency key.

See idempotency for the general pattern and data pipeline best practices for how it applies across a whole pipeline.

What are backfills and catchup?

A backfill runs a DAG for past intervals, for example after fixing a bug or adding a new table. When catchup=True, the scheduler automatically creates runs for every missed interval between start_date and now. Many teams set catchup=False and trigger backfills deliberately, so a new DAG does not launch hundreds of runs the moment it is deployed.

In Airflow 2 you can backfill from the CLI with airflow dags backfill, passing a start and end date. Airflow 3 moved backfills into the scheduler and exposes them through the UI, API, and the airflow backfill create command. Either way, backfills only work well when tasks are interval-scoped and idempotent.

Airflow best practices

1. Keep DAG files light. The scheduler parses them often, so avoid database calls or heavy imports at the top level.
2. Let Airflow orchestrate and push heavy processing to engines built for it, such as Spark, the warehouse, or a container.
3. Keep tasks small and focused so a failure reruns only the part that broke.
4. Store credentials in connections or a secrets backend, never in DAG code.
5. Set timeouts and SLAs or deadline alerts so stuck tasks are noticed.
6. Test DAGs in CI by at least importing them and checking for cycles and import errors.

Key takeaways

• Airflow orchestrates pipelines defined as Python DAGs; it coordinates work rather than processing data itself.
• The scheduler decides what runs, the executor decides where, and the metadata database records everything.
• Use data intervals, not the current time, to scope each run.
• Retries and backfills are only safe when tasks are idempotent.
• Pass small references through XCom and keep heavy computation outside the Airflow workers.

Frequently asked questions

Is Airflow an ETL tool?

Not exactly. Airflow is an orchestrator that schedules and monitors ETL or ELT steps, but the extraction, loading, and transformation usually happen in other systems it triggers. For background on those patterns, see ETL vs ELT.

Can Airflow handle streaming data?

Airflow is designed for batch workflows that run on a schedule or in response to events such as a dataset update. It is not a stream processor for continuous, low-latency events. Use a streaming system for that and let Airflow orchestrate the surrounding batch jobs.

What is the difference between the TaskFlow API and classic operators?

TaskFlow uses decorated Python functions and passes return values between tasks automatically, which reduces boilerplate. Classic operators are instantiated as objects and wired with >>. Both run on the same engine, and you can mix them in one DAG.

Should I set catchup to True or False?

Set catchup=False unless you specifically want the scheduler to create runs for every missed interval. Turning it off prevents a burst of historical runs when you deploy or unpause a DAG. You can still backfill specific date ranges on purpose when you need them.