Why Evals in Python

A sentence in a prompt is not an if that one message goes through. The model reads the whole prompt with every message. So a sentence you add to fix one answer can change other answers too, and nothing tells you.

A small example. A spam filter lets an ad through, so you add one sentence to its prompt: If the message has a link, answer spam. You try the ad again and get spam. Fixed. But a customer who pasted a tracking link is now spam too, and you never looked at that message.

The way to see such damage: keep the old prompt as "v1", add the new one as "v2", run both on every message and print what changed. The Corner Shop's assistant sorts messages into refund, shipping, product or other, and this lesson's program must print:

v1: shipping
v2: refund
CHANGED: My lamp arrived broken. I want my money back. | shipping -> refund
same: Where is my package? I ordered ten days ago. | shipping
CHANGED: How much does shipping to Canada cost? | shipping -> refund
CHANGED: Is the teapot worth the money? | product -> refund
same: Do you have a job for a student this summer? | other
3 of 5 answers changed

Three answers changed. One is the fix you wanted. The other two are messages that were right before. Reading every line works for five messages, not for five hundred. That is what evals are for: a fixed set of inputs, the answer you expect for each one, and a program that checks them all after every change.

Your task: the prompts, the inbox and the function classify(message, version) are already there. Add the prompt "v2", try it on the first message, then loop over the inbox: ask both versions, print same: or CHANGED: for each message, and print how many answers changed.

Assignment
Follow the Coding Tutorial to add prompt v2, compare it with v1 on all five messages and count the answers that changed.

Hint
Every message needs two calls to classify, one per version. changed = 0 goes above the loop, the else: lines up with the if, and the last print sits at the left edge, after the loop.