Chunk Size Trade-offs in Python

Until now a chunk has been one paragraph. It could also be a whole file, or a single sentence. Big chunks mix many topics in one vector and cost more tokens. Small chunks lose their context. You don't have to guess which size is best: you can measure it.

To measure, you need test questions that each come with the file where the answer is. For every question you search, and you count a hit when that file is among the top three chunks. The model can only answer from a passage it was given.

A function as a parameter: one function can test any way of splitting, if the splitting function is handed to it. A function goes in by its name alone, without parentheses:

def shout(text):
    return text.upper() + "!"

def show(change):
    print(change("hello"))

show(shout)    # HELLO!

When your program is finished, it prints one row per chunk size:

file 9 chunks, 4 of 5
paragraph 27 chunks, 5 of 5
sentence 45 chunks, 4 of 5

Your task: the starting code already has build_index(split), top_matches(question, chunks) and the five TESTS. Write split_file (one chunk per file) and split_sentences (one chunk per sentence). Then write measure(name, split), which counts the hits for one size and prints its row, and call it for the three sizes.

Assignment
Follow the Coding Tutorial to write the two new splitters and measure, and to print the table for the three chunk sizes.

Hint
split_sentences needs no loop of its own: put an empty line after every sentence and let split_document do the cutting. A splitting function goes to measure by its name alone, the way by_score goes to sorted.