The Assistant End to End, 5: Shipping It in Python

The assistant passed its gate. To ship it, you put it behind a web address: the shop's website sends a question to POST /chat and gets the answer back as JSON. The function answer and everything it needs are already in the program.

A shipped app has to live through a busy model. A retry tries the same call again after a short wait; the SDK's client does that by itself, and client.with_options(max_retries=...) sets how many times. When a model fails on every try, a fallback asks another model instead.

A cache keeps answers that were already worked out, so the same question doesn't cost time and money twice. Here is the idea on another value, a slow lookup of delivery prices:

prices = {}

def price_for(country):
    if country in prices:
        return prices[country]
    prices[country] = slow_lookup(country)
    return prices[country]

Cache only what stays true. The shipping policy is the same tomorrow; an order's status is not. So the assistant stores only the answers that used no tools.

These are the requests the tests send, and what your app must answer:

POST /chat {"question": "Do you ship to Canada?"}  →  200  {"answer": "Yes, we ship to Canada. …", "sources": ["shipping.md"], "tools": []}
POST /chat {"question": "do you ship to canada"}  →  200  the same answer, from the cache
POST /chat {"question": "Can you look up order 5 for me?"}  →  200  {"answer": "… It is paid and has not shipped yet.", "sources": [], "tools": ["lookup_order"]}
POST /check/ship-order-5  →  200  {"order": 5, "status": "shipped"}
POST /chat {"question": "Can you look up order 5 for me?"}  →  200  {"answer": "Order 5 was placed on 2026-05-07 and it has shipped.", …}
POST /chat {"question": "Can I get my money back?"}  →  200  {"answer": "Yes. You can return any item within 14 days of delivery, …", "sources": ["returns.md"], "tools": []}
GET /check/model  →  200  13 model calls: 12 to claude-haiku-5-5, then 1 to claude-sonnet-5-5
GET /check/log  →  200  every call with its example cost, one cache hit, one warning

In these tests the model is busy on purpose: its first three calls answer 429 (too many requests), and the small model is overloaded (529) every time it is asked about money back.

Your task: write the chat function and give it the address POST /chat. Then let the client retry 3 times, add the fallback model in ask_model, add the cache to chat (only for answers that used no tools), and log the cost of every model call and every cache hit.

Assignment
Follow the Coding Tutorial to serve the assistant at POST /chat with retries, a fallback model, a cache and cost logging.

Hint
The decorator goes on the line right above def chat():. Inside chat the order matters: read the question, make the key, look in the cache, call answer, store the result if its "tools" list is empty, return. Run the tests after every step and read /check/model and /check/log: they show what really reached the model.