When the Chat Gets Too Long in Python

A model keeps nothing between calls, so every call sends the whole chat again: the system prompt and every message so far. A chat only grows, and every token costs money and time, on every call. A model can also read only so much in one call: its context window.

So a chat gets a token budget, and you check its size before you send it. client.messages.count_tokens takes the same model, system and messages as create, but the model writes nothing: you only get a count back, in .input_tokens.

When the chat is over budget, the oldest turns go first, because they matter least to the newest question. pop(0) removes the first item of a list:

queue = ["Ana", "Dan", "Ioana", "Mara"]
queue.pop(0)
queue.pop(0)
print(queue)   # ['Ioana', 'Mara']

A turn is a pair: the customer's message and the answer to it. Remove both and the list still starts with a user message, as the API wants. The system prompt is not in the list, so it is never dropped.

With a budget of 100 tokens, your finished program prints this (the last three lines come from the check, which looks at what the model received):

Tokens before: 146
Tokens after: 83
Messages kept: 5
Shop: If you order today, the mug reaches Canada in 7 to 10 days.
Input tokens: 83
Messages the model received: 5
The first one is from: user
System prompt received: yes

Your task: the chat, the BUDGET and the call that answers the customer are already written. Write a helper count_tokens(messages), measure the chat before it is sent, drop the oldest turns in a loop until the chat fits the budget, and print the tokens before and after and the number of messages kept.

Assignment
Follow the Coding Tutorial to count the chat's tokens before sending it and to drop the oldest turns until it fits the budget.

Hint
The loop must count again after every turn it drops. If tokens never changes inside the loop, the loop only stops when one message is left.