Remember Where You Stopped in Python

A pipeline that runs every day must know where it stopped. If each run loads only "yesterday", a day whose run never happened (the server was down, the run failed) is skipped for good, and nobody is told. If each run loads everything, the runs get longer every week. A state file fixes both: a small file where the pipeline writes the last day it loaded, and reads it back at the start of the next run.

Here the state is state.json, one JSON object such as {"last_loaded": "2026-05-04"}. The starting code reads it with json.load in read_state and writes it with json.dump in write_state. On the very first run the file isn't there and open stops with FileNotFoundError, so you ask first with os.path.exists(path), which gives True or False, and return a default date when the file is missing.

Text to date: the state holds text, but to add a day you need a date. date.fromisoformat turns "2026-05-31" into a date, the opposite of .isoformat(), and a date plus timedelta(days=1) rolls over the month's end:

day = date.fromisoformat("2026-05-31") + timedelta(days=1)
print(day.isoformat())   # 2026-06-01

With run_date = "2026-05-04", the script calls main() twice. The first call loads 05-01 to 05-04 and saves the state; the second finds nothing to do:

Last loaded: 2026-04-30
Loaded 2026-05-01: 10 rows
Loaded 2026-05-02: 12 rows
Loaded 2026-05-03: 10 rows
Loaded 2026-05-04: 9 rows
Last loaded: 2026-05-04

Then the check sets run_date to "2026-05-06" and calls main(): it must load only 05-05 and 05-06. Schedulers such as Airflow keep this record for you in their own database; the idea is the same.

Your task: in read_state, return "2026-04-30" when state.json doesn't exist. In main, set start to the day after last, call backfill from start to run_date, then save run_date with write_state. At the bottom, call main() a second time.

Assignment
Follow the Coding Tutorial to make the pipeline read its state, load only the new days, and save the state.

Hint
Save the state last, after backfill: if a day fails, the state still points to the last good day, and the next run tries again from there. backfill needs two dates, so turn run_date into one too.