Choosing a Model in Python

A model family comes in several sizes. The small model is fast and cheap. The bigger ones reason better, answer more slowly and cost more. Which one should the shop use? You don't guess: you ask them all the same question and compare the answers.

To compare, your program needs the models with their prices, and a check it can run itself. A loop over a dictionary gives its keys one by one, so a dictionary of prices is also a list of names:

sizes = {"small": 2, "large": 9}
for name in sizes:
    print(name, "costs", sizes[name])
# small costs 2
# large costs 9

In this lesson EXAMPLE_PRICES holds three model names with a price each, in dollars per million input tokens. They are example prices: real ones change, so look them up on the provider's pricing page. The question is a trap: a mug and a pen cost 14 dollars together, and the mug costs 12 dollars more than the pen. It ends with "Answer with the number only", so the right answer is exactly the string in EXPECTED. The course's model plays all three models, and in this lesson the smallest one falls for the trap.

Your program must print:

claude-haiku-5-5 answers 2
claude-sonnet-5-5 answers 1
  correct, at an example price of 3
claude-opus-5-5 answers 1
  correct, at an example price of 5
Use: claude-sonnet-5-5

Your task: the client, EXAMPLE_PRICES, EXPECTED, messages and best = None are already written. Loop over the models, ask each one the question and print its answer. When the answer equals EXPECTED, print the model's example price and keep the model in best if it is the cheapest one that passed so far. After the loop, print best.

Assignment
Follow the Coding Tutorial to ask three models the same question and pick the cheapest one whose answer passes the check.

Hint
Only the model argument changes from call to call. The price comparison sits inside the check, so a model with a wrong answer is never chosen, however cheap it is. The line with two spaces before correct must match exactly.