Start learning free

LESSON 43 OF 107 · DATA ENGINEERING: DATA QUALITY

Freshness: Is the Data Late? in Python

Practice

The checks so far looked at the rows that arrived. Freshness asks whether the newest rows arrived at all: take the newest ordered_at in the table and count the days to the run date. When an export stops, the rows you already have stay clean and every row count still looks normal, because a file that never came adds no rows and no line to GROUP BY filename. A data engineer checks freshness so that a late feed stops the run, instead of a report quietly showing old numbers as if they were today's.

New SQL: date_diff('day', a, b) counts the days from date a to date b, so the order matters: the older date goes first. CAST(ordered_at AS DATE) turns the text "2026-05-01 08:25:00" into the date 2026-05-01, and MAX keeps the newest. A ? in a query is a parameter: con.execute(query, [value]) puts the value in its place, so one query works for any run date. An example on another value:

query = (
    "SELECT COUNT(*) "
    "FROM raw_orders "
    "WHERE status = ? "
)
cancelled = con.execute(query, ["cancelled"]).fetchone()[0]

The starting code has run_date = "2026-05-09": the pipeline runs on the morning of May 9, so it should have orders up to May 8. The week's files stop at 2026-05-07: the 05-08 export never came. Data more than 1 day old fails. The hidden check then prints a heading and calls your function for a run on 2026-05-08, when orders from 05-07 are right on time. The program prints:

orders are fresh: FAIL (2 days old)
Run on 2026-05-08:
orders are fresh: PASS

Your task: inside check_freshness(run_date), write the query that gives the days from the newest order day to the run date, run it with the run date for the ? and keep the number in days_old, then print orders are fresh: FAIL (N days old) when it is more than 1, or orders are fresh: PASS otherwise.


Assignment
Follow the Coding Tutorial to write the freshness check and print its PASS or FAIL line.


Hint
In date_diff, put the newest order day first and the ? second: swapped, the days come out negative and the check can never fail. fetchone()[0] takes the one value of the one row.


STEP 1 OF 7
    Ctrl + Enter to check

    De Dq Freshness in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free