Start learning free

LESSON 32 OF 107 · DATA ENGINEERING: PIPELINES IN PYTHON

Backfill a Range of Days in Python

Practice

A backfill runs a pipeline for a range of past days in one go. You need one when a new pipeline must load the history, and after every bug fix, because the days loaded with the old code must be loaded again. Typing one run("…") per day by hand skips days and breaks at month ends.

Python's datetime module does the date arithmetic. date(2026, 5, 1) makes a date. Adding timedelta(days=1) gives the next day, across month and year ends. Two dates compare like numbers: an earlier day is smaller. .isoformat() turns a date into text like "2026-05-01", which is what run builds its paths from:

from datetime import date, timedelta
d = date(2026, 4, 30)
d + timedelta(days=1)        # date(2026, 5, 1)
d < date(2026, 5, 1)         # True
d.isoformat()                # '2026-04-30'

The pipeline from the last lessons is written, and the hidden setup has already loaded 05-01 and 05-02. Your backfill loads 05-01 to 05-04, both ends included. Because load deletes a day's folder before writing it, the days already there come out the same, not doubled. 05-03 keeps 10 of its 12 rows: transform drops the second copy of order 5027 and order 5030, whose quantity is "two".

Then the check calls its own backfill, from 05-03 to 05-05:

date | orders | first_order | last_order
2026-05-01 | 10 | 5001 | 5010
2026-05-02 | 12 | 5011 | 5022
2026-05-03 | 10 | 5023 | 5033
2026-05-04 | 9 | 5034 | 5042
(4 rows)
The check calls backfill(date(2026, 5, 3), date(2026, 5, 5)):
date | orders | first_order | last_order
2026-05-01 | 10 | 5001 | 5010
2026-05-02 | 12 | 5011 | 5022
2026-05-03 | 10 | 5023 | 5033
2026-05-04 | 9 | 5034 | 5042
2026-05-05 | 13 | 5040 | 5054
(5 rows)

Your task: write backfill(start, end): a while loop walks a day from start to end and calls run for each one. Then call it from date(2026, 5, 1) to date(2026, 5, 4).


Assignment
Follow the Coding Tutorial to write backfill and load 05-01 to 05-04 with it.


Hint
Type the line that moves day forward before the run call: a while loop whose day never changes never ends. Use <= so the end day is loaded too.


STEP 1 OF 6
    Ctrl + Enter to check

    De Pipe Backfill in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free