Retries in Python

Some failures fix themselves. A source is busy for a second, the network drops out, a database restarts: run the task again a moment later and it works. An orchestrator retries a failed task: it runs it again, up to a set number of times, and marks it "failed" only when the last try fails too. Without retries, one short timeout at 3 a.m. stops the whole night's pipeline, and someone reruns it by hand in the morning.

retries = 2 means 2 more tries after the first one, so a task gets at most 3 tries. The runner counts them in a dict, attempts: task name to number of tries. A real bug, like the quantity "two", fails on every try: retries don't fix it, they only use up 3 tries before the task is marked failed. In Airflow you set retries=2 on a task, usually with a retry_delay, a wait between tries; this lesson's runner doesn't wait.

The retry loop is a while with and: it goes on only while both tests are true, so it stops at whichever turns false first. Here a call is tried until it stops answering "busy", but never more than 3 times:

answers = ["busy", "busy", "ok"]
tries = 0
reply = "busy"
while reply == "busy" and tries < 3:
    reply = answers[tries]
    tries = tries + 1
print(reply, "after", tries, "tries")    # ok after 3 tries

In this lesson's code, extract_orders raises TimeoutError on its first two calls and works on the third. Right now the runner tries each task once, so extract_orders fails and the three tasks after it are upstream_failed. Your finished code must print:

extract_orders failed: no answer from the orders source
extract_orders failed: no answer from the orders source
extract_orders done
extract_customers done
clean_orders done
build_report done
send_email done
extract_orders: success | attempts: 3
extract_customers: success | attempts: 1
clean_orders: success | attempts: 1
build_report: success | attempts: 1
send_email: success | attempts: 1

Then the check makes clean_orders fail on every call: its error must print 3 times, it must end clean_orders: failed | attempts: 3, and build_report and send_email must end upstream_failed | attempts: 0.

Your task: set retries to 2. In run_pipeline, record 1 attempt before the first run_once call; then, while the task's status is "failed" and its attempts are at most retries, add 1 to its attempts and run it again with run_once.

Assignment
Follow the Coding Tutorial to give the runner retries: count each task's tries and run a failed task again while it has retries left.

Hint
The first try is already in the code. With attempts at 1 and the test attempts[name] <= retries, the loop runs for tries 2 and 3, then stops. Add 1 to the attempts before you call run_once again, so the count always matches the tries.