Backfills in Python

Sometimes a day gets no run: the pipeline was down, or a run crashed. Running the pipeline later for past run dates is a backfill. The orchestrator keeps a record of finished runs and runs only the dates of the range that are missing from it, oldest first. Without that record, a backfill runs every day again: finished reports are rebuilt, emails go out twice, and a long range takes hours for nothing.

A set as the record: a Python set is written in curly brackets, like a dict without values. It holds each value at most once. x in s is True when the set holds x, and s.add(x) puts x in.

seen = {"2026-04-01"}
print("2026-04-02" in seen)   # False
seen.add("2026-04-02")
seen.add("2026-04-02")
print(len(seen))              # 2: a set holds a value once

Inside a loop, continue skips the rest of the current turn and goes on with the next value. A date goes into the record only after its run: if the run crashes, the date stays missing, and the next backfill tries it again.

The starting code has the pipeline, daily_run_dates from the last lesson, and a backfill(start, end) that runs every date of the range. The program backfills May 1 to May 4 twice. May 1 and May 3 already ran; the pipeline was down on May 2, and May 4 hasn't run yet. Once the record works, the program prints this (the check adds the last line):

First backfill:
Run for 2026-05-02
extract_orders reads orders/2026-05-02.csv
extract_customers done
clean_orders done
build_report writes report_2026-05-02.csv
send_email done
Run for 2026-05-04
extract_orders reads orders/2026-05-04.csv
extract_customers done
clean_orders done
build_report writes report_2026-05-04.csv
send_email done
Second backfill:
Finished runs: ['2026-05-01', '2026-05-02', '2026-05-03', '2026-05-04']

Airflow keeps this record in its database, and it backfills when you ask it to (or by itself, with catchup=True). Here you build it small with a set.

Your task: make the set finished with May 1 and May 3, skip every run date that is in it, and add each run date to it after its run.

Assignment
Follow the Coding Tutorial to give backfill a record of finished runs: a set, an if with continue, and finished.add.

Hint
The if and its continue go at the top of the loop, before run_pipeline. The date goes into finished below the run_pipeline call, still inside the loop.