Dropping the oldest turns keeps a chat short, but it saves tokens by forgetting. If the customer said their order number in the first message and that message is gone, the assistant has to ask for it again.
A better way is to summarize: the old turns are replaced by a few lines that say what mattered in them. A model writes the summary, in a call of its own. Here is the idea on another chat:
user: Hi, I'm Dan. A table for four on Saturday, please. assistant: Booked: Saturday at 19:00, four people. user: Can we sit outside? assistant: Yes, I moved you to the terrace. Summary: Dan has a table for four on Saturday at 19:00, on the terrace.
The summary call gets its own system prompt, and that prompt says what the summary must keep: here, order numbers, product names and dates. The summary then goes into the system prompt of the call that answers the customer, and the newest messages still travel word for word.
Your finished program prints:
Whole chat: 144 tokens Summary: Order 1042 is one Teapot, shipped on Monday and due to arrive by Friday. The Teapot is dishwasher safe, and the matching Mug costs $12. Shop: Your order 1042 should arrive by Friday. Input tokens: 92 Model calls: 2 Messages in the last call: 3
Your task: the starting code keeps the last KEEP messages and throws the older ones away. Keep the removed messages in old, join them into one transcript, ask the model for a summary with the SUMMARIZE system prompt, print the summary, and add it to the system prompt of the call that answers the customer.
Assignment
Follow the Coding Tutorial to replace the old turns of the chat with a summary written by the model.
Hint
There are two calls of client.messages.create. The first gets SUMMARIZE and the transcript; the second gets the shop's rules with the summary added, and the last three messages. The text of a reply is in .content[0].text.
Free plan: 30 tutor messages. PRO is unlimited.
