Cache Model Answers in Python

Interview question: "Your support bot gets the same questions all day, and every model call costs money and a second or two. Add a cache."

A response cache is a dict from a request to its answer. The key must hold everything that changes the answer: the model, the system prompt and the messages. Turn them into one text with json.dumps(..., sort_keys=True), so equal requests always give equal text. Then hash that text with hashlib.sha256 to get a short key of fixed length. Look the key up before calling the model; on a miss, call it and store the answer.

The same idea on other data: a translation helper could key on the language and the sentence. hashlib.sha256(json.dumps(["fr", "Good morning"]).encode()).hexdigest() gives the same 64 characters on every run, and changing either part gives a new key.

This lesson's script asks "Do you ship to Canada?" four times. The second ask repeats the first, the third adds " Be friendly." to the system prompt, and the fourth uses another model. Only the repeat is a hit:

Yes, we ship to Canada in 7 to 10 days.
Yes, we ship to Canada in 7 to 10 days.
Yes, we'd love to ship to Canada! Your parcel arrives in 7 to 10 days.
Yes: delivery to Canada takes 7 to 10 business days and costs $15.
Model calls: 3
Cache entries: 3

Your task: write cache_key(model, system, messages) and use it in ask: compute the key, store each new answer in cache, and return the stored answer on a hit, before any call to the model.

Assignment
Follow the Coding Tutorial to build the cache key and the lookup that skips repeated calls.

Hint
sort_keys=True writes every dict's keys in one order, so two equal messages give the same key. A hash needs bytes: call .encode() on the text first.