Project: A Name Generator, and What the Real Ones Add in Python

Your TinyGPT is ready. The starting code holds every part you built in this chapter, trains the model for 400 steps on names.txt, and has generate(temperature), which gives back one name the model writes, as a string. One question is left: does the model make up names, or does it only give back the ones it was trained on?

Copies and new names: a sample is a copy if it is already in training_names, the list of the file's 600 names that text.split() cuts out of the text. Python's in looks through a list and gives back True or False; not in gives back the opposite. With not in you can keep each value only once:

seen = []
for colour in ["red", "blue", "red"]:
    if colour not in seen:
        seen.append(colour)
print(seen)   # ['red', 'blue']

Let the model write 100 names at temperature 1.0, which keeps its own probabilities. Count the copies, and put every other name in new_names the first time it comes out. Make the second check an elif, so a copy is never kept as new. The program must print:

step 0 loss: 2.986
step 100 loss: 1.446
step 200 loss: 1.431
step 300 loss: 1.392
copies of training names: 52 of 100
10 new names: ['sulope', 'bami', 'niza', 'tada', 'roza', 'motaba', 'mikaza', 'tale', 'rotu', 'nirimo']

About half the samples are copies. The check then confirms that none of the 10 new names is in names.txt, and that the 100 samples held 40 different new names.

What the real ones add: the models behind ChatGPT are built from the same parts: embeddings, attention blocks, and a last layer that scores the next token. Three things set them apart. Word-piece tokens: a vocabulary of 50,000 or more pieces, learned with byte-pair merges, so one position holds a chunk like ing and far more text fits in the context. Scale: dozens of blocks instead of 2, billions of weights instead of about 27,000, contexts of thousands of tokens instead of 8 characters, and trillions of tokens of text instead of 600 names. Fine-tuning: after learning to continue text, the model is trained further on example conversations and on people's ratings of its replies. That is what turns a text continuer into an assistant that answers you.

Your task: below training_names, set copies to 0 and new_names to an empty list. Then loop 100 times: store generate(1.0) in name; if it is in training_names, add 1 to copies; otherwise, if it is not yet in new_names, append it.

Assignment
Follow the Coding Tutorial to count how many of 100 generated names are copies of names.txt, and to keep each new name once.

Hint
Both checks go inside the loop: an if for the copies, then an elif for the new names. With two separate ifs, a copy would land in new_names too.