When a Job Fails in Python

The worker from the last lesson marks every job done, whatever the mail service answered. But a busy mail service answers 503 Service Unavailable ("too busy, try later"), and then the email never left. The job says done, so no run will try it again. A job that fails should stay 'waiting', so the next run retries it.

Not forever, though: an email to an address that doesn't exist fails at every run. So each job counts its attempts, in a new column attempts INTEGER NOT NULL DEFAULT 0. DEFAULT 0 fills it in for new jobs, so the INSERT in pay_order stays as it is. After the third failed attempt the job is 'failed': the worker leaves it alone, and a person can look at it.

Counting tries looks the same everywhere. A login that locks after 5 wrong passwords adds 1 first, then compares:

tries = user["wrong_passwords"] + 1
if password_ok:
    state = "open"
elif tries < 5:
    state = "try again"
else:
    state = "locked"

The mail service answers 202 when it takes an email. In the tests, POST /check/fail-next makes it answer 503 to the next 4 emails, then it works again:

POST /orders/8/pay                  →  201  {"amount": 80, "order_id": 8, "payment_id": "pay_1"}
POST /check/fail-next {"times": 4}  →  200  {"failing": 4}
POST /admin/run-jobs                →  200  {"ran": 1}
POST /orders/10/pay                 →  201  {"amount": 21, "order_id": 10, "payment_id": "pay_2"}
POST /admin/run-jobs                →  200  {"ran": 2}
GET /check/jobs                     →  200  [{"attempts": 2, "id": 1, "order_id": 8, "status": "waiting"},
                                             {"attempts": 1, "id": 2, "order_id": 10, "status": "waiting"}]
POST /admin/run-jobs                →  200  {"ran": 2}
GET /check/jobs                     →  200  [{"attempts": 3, "id": 1, "order_id": 8, "status": "failed"},
                                             {"attempts": 2, "id": 2, "order_id": 10, "status": "done"}]
GET /check/sent                     →  200  [{"subject": "Your order 10 is paid", "to": "chloe@example.com", …}]
POST /admin/run-jobs                →  200  {"ran": 0}

Your task: the jobs table's line is gone from your code: write it again with the attempts column. In the worker's loop, keep the answer of send_receipt, add 1 to the job's attempts, and choose the status: "done" after a 202, "waiting" while attempts is below 3, "failed" otherwise. Save the status and the attempts with one UPDATE.

Assignment
Follow the Coding Tutorial to count each job's attempts and retry a failed job up to 3 times.

Hint
Check the 202 first, so a job that got through is never sent again. Add 1 to the attempts before you compare with 3: the third failure must already count as 3.