Project: The Daily Orders Pipeline in Python

This project puts the chapter together into one pipeline. It loads the whole week of order exports, orders/2026-05-01.csv to orders/2026-05-07.csv, into one Parquet folder per day, warehouse/orders/date=<day>/. The starting code already has extract, an empty transform, a load that adds the new rows to what a day's folder already holds, a run that sends each row to the folder of the day in its ordered_at, and backfill. You add what makes the week's numbers right.

The week's trouble: 05-03 has the quantity "two" on order 5030, and order 5027 twice. 05-04 brings order 5036 late (it was placed on 05-02). 05-05 repeats order 5040. 05-06 has no customer_id on order 5058. 05-07 sends order 5062 again, now cancelled.

Three cleaning tools: pd.to_numeric(column, errors="coerce") turns text into numbers. Text that isn't a number becomes NaN (a missing value) instead of stopping the run. dropna(subset=[…]) drops a row when any of the listed columns is missing. drop_duplicates("order_id") keeps one row per order id, and keep="last" keeps the newest copy. Without them, one bad cell stops the night's load, or an order sent twice is counted twice.

signups = pd.DataFrame({"user": [7, 7, 9, None], "age": ["31", "31", "n/a", "40"]})
signups["age"] = pd.to_numeric(signups["age"], errors="coerce")  # "n/a" becomes NaN
signups = signups.dropna(subset=["user", "age"])  # drops user 9 and the row with no user
signups = signups.drop_duplicates("user", keep="last")  # one row left: user 7, age 31

Log and report: each run prints one log line with the date and how many rows it read, kept and dropped. A day that loses rows, or reads none, then stands out at once. The query reads every folder with read_parquet('warehouse/orders/*/*.parquet', hive_partitioning = true): hive_partitioning turns the folder name date=2026-05-01 into a date column. It counts the paid orders and sums quantity * unit_price per date. Your runs print the log lines, then the check prints your query's result:

2026-05-01 read 10 kept 10 dropped 0
2026-05-02 read 12 kept 12 dropped 0
2026-05-03 read 12 kept 10 dropped 2
2026-05-04 read 9 kept 9 dropped 0
2026-05-05 read 13 kept 13 dropped 0
2026-05-06 read 10 kept 9 dropped 1
2026-05-07 read 12 kept 12 dropped 0
date | orders | revenue
2026-05-01 | 10 | 274
2026-05-02 | 11 | 384
2026-05-03 | 8 | 245
2026-05-04 | 6 | 122
2026-05-05 | 12 | 436
2026-05-06 | 6 | 168
2026-05-07 | 9 | 595
(7 rows)

The check then prints the warehouse totals (73 rows, 73 orders, 2224 of paid revenue), runs 05-07 a second time and prints the same totals again: the pipeline can run again safely.

Your task: print the log line at the end of run, call backfill from date(2026, 5, 1) to date(2026, 5, 7), clean the rows in transform (numbers, missing values, repeats), keep the last copy of each order in load, and finish the query with its SELECT line and a WHERE for paid orders.

Assignment
Follow the Coding Tutorial to finish the daily orders pipeline: log each run, backfill the week, clean each day's rows, keep each order once and report paid orders and revenue per day.

Hint
Write the cleaning lines in this order: to_numeric first, so "two" is already NaN when dropna looks for missing values, then drop_duplicates. In the query, the WHERE line goes between FROM and GROUP BY.