The Assistant End to End, 4: Evals in Python

The assistant has knowledge, tools and guardrails. Does it still work after you change one line? So far you have read a few answers by eye. An eval does the reading for you: a fixed list of questions (the test set), a check for each one, and a program that runs them all after every change.

Each case holds an input and what you expect back. One case for another program, a calculator bot, looks like this:

case = {"input": "2 + 2", "expected": "4"}
result = calculator(case["input"])
print(result == case["expected"])   # True or False

The assistant's answers are free text, and the model may word them differently tomorrow. So each kind of case checks a fact instead. A "policy" case passes when the expected document is in the result's sources. An "order" case passes when the expected tool is in its tools. A "refuse" case passes when the answer is exactly the fixed sentence.

The score is how many cases passed. A gate turns it into a decision: under 90% the program ends with sys.exit(1), and a deploy script that reads the exit code stops there. Your first run finds a real bug and scores 8 of 9. After you fix the prompt, the program must print:

Blocked: a discount offer
Blocked: too many tool calls
Score: 9 of 9
GATE: PASS

The check below the program then tries passes on six results of its own: for each kind, one that must give True and one that must give False.

Your task: the assistant and the list cases are already there. Write the scorer passes(case, result) for the three kinds, run every case through answer, print a FAIL line for each case that fails, print the score, add the gate, and then fix the assistant so that all nine cases pass.

Assignment
Follow the Coding Tutorial to write the scorer, run the nine cases, print the score, add the gate and fix the prompt.

Hint
An in test and == already give True or False, so passes can return them directly. The line that prints GATE: PASS sits at the left edge, after the if: the program gets there only when it did not exit. When a case fails, fix the assistant, not the test.