Text into Numbers in Python

A language model, like the one behind ChatGPT, only works with numbers: it multiplies and adds them. So every model starts with a tokenizer, which turns text into a list of numbers called ids and turns ids back into text. In this chapter you build a tiny model that invents names, one part per lesson. This is part one.

The simplest tokenizer gives each character its own id. The vocabulary is the list of the different characters, each once, in sorted order, and a character's id is its position in that list. set(text) gives back the different characters of a string, and sorted(...) gives them back as a list in order. On the word "banana":

sorted(set("banana"))   # ['a', 'b', 'n']
# a has id 0, b has id 1, n has id 2
# so "banana" encodes to [1, 0, 2, 0, 2, 0]

Your file names.txt holds made-up names, one per line. The newline "\n" that ends each name is a character too: it sorts before the letters, so it gets id 0. The names use only 18 letters, so the vocabulary has 19 characters. Real models use pieces of words as tokens instead of single characters, but the idea is the same.

When your code is done, the checks that run after it print:

vocabulary size: 19
id of the newline: 0 id of a: 1 id of z: 18
encode('rono'): [13, 11, 10, 11]
decode back: rono
decode([7, 1, 9, 6, 0]): 'kami\n'
decoded back to the same text: True

Your task: the starting code reads the file into text and gives you an empty dictionary stoi (string to id) and two unfinished functions. Store the vocabulary in chars, fill stoi with each character's id, finish encode(s) so it gives back the list of ids of a string, and finish decode(ids) so it gives back the string.

Assignment
Follow the Coding Tutorial to build the vocabulary, the stoi dictionary, encode and decode.

Hint
An id is a position in chars: stoi[ch] turns a character into its id, and chars[i] turns id i back into its character.