When a Job Fails in C#

The worker from the last lesson marks every job done, whatever the mail service answered. But a busy mail service answers 503 Service Unavailable ("too busy, try later"), and then the email never left. The job says done, so no run will try it again. A job that fails should stay 'waiting', so the next run retries it.

Not forever, though: an email to an address that doesn't exist fails at every run. So each job counts its attempts, in a new column attempts INTEGER NOT NULL DEFAULT 0. DEFAULT 0 fills it in for new jobs, so the INSERT in the pay route stays as it is. After the third failed attempt the job is 'failed': the worker leaves it alone, and a person can look at it.

Counting tries looks the same everywhere. A login that locks after 5 wrong passwords adds 1 first, then compares. In C#, a variable declared without a value, like state, gets its value in each branch:

var tries = Convert.ToInt32(user["wrong_passwords"]) + 1;
string state;
if (passwordOk) {
    state = "open";
} else if (tries < 5) {
    state = "try again";
} else {
    state = "locked";
}

The mail service answers 202 when it takes an email. In the tests, POST /check/fail-next makes it answer 503 to the next 4 emails, then it works again:

POST /orders/8/pay                  →  201  {"amount": 80, "order_id": 8, "payment_id": "pay_1"}
POST /check/fail-next {"times": 4}  →  200  {"failing": 4}
POST /admin/run-jobs                →  200  {"ran": 1}
POST /orders/10/pay                 →  201  {"amount": 21, "order_id": 10, "payment_id": "pay_2"}
POST /admin/run-jobs                →  200  {"ran": 2}
GET /check/jobs                     →  200  [{"attempts": 2, "id": 1, "order_id": 8, "status": "waiting"},
                                             {"attempts": 1, "id": 2, "order_id": 10, "status": "waiting"}]
POST /admin/run-jobs                →  200  {"ran": 2}
GET /check/jobs                     →  200  [{"attempts": 3, "id": 1, "order_id": 8, "status": "failed"},
                                             {"attempts": 2, "id": 2, "order_id": 10, "status": "done"}]
GET /check/sent                     →  200  [{"subject": "Your order 10 is paid", "to": "chloe@example.com", …}]
POST /admin/run-jobs                →  200  {"ran": 0}

Your task: some lines are gone from your code, so it doesn't compile yet (done doesn't exist). Write the jobs table's line again with the attempts column. In the worker's loop, keep the answer of SendReceipt in response, add 1 to the job's attempts, and choose a string status: "done" after a 202, "waiting" while attempts is below 3, "failed" otherwise. Then write the done command again, so one UPDATE saves the status and the attempts, and give it both values.

Assignment
Follow the Coding Tutorial to count each job's attempts and retry a failed job up to 3 times.

Hint
Check the 202 first, so a job that got through is never sent again. Add 1 to the attempts before you compare with 3: the third failure must already count as 3. The code compiles again once the done line is back.