Guess the Next Letter by Counting in Python

A language model does one job over and over: given the text so far, it gives every possible next character a probability, then draws one. Before any learning, you can get those probabilities by counting. This lesson builds a bigram model: it looks only at the last character.

Count how often each character is followed by each other one, in a table with one row per character and one column for the character that comes next. Then divide each row by its own total, so it adds up to 1. In the word papaya, a is followed by p once and by y once, so after a, p and y each get 0.5. torch.zeros(rows, columns) gives back a table full of 0.0, and t.sum(dim=1, keepdim=True) gives back one total per row, as a column:

t = torch.tensor([[1., 3.], [2., 2.]])
t.sum(dim=1, keepdim=True)   # [[4.], [4.]]
# divided by it, the rows become [0.25, 0.75] and [0.5, 0.5]

To invent a name, start at the newline, draw the next character from its row, then from that character's row, until the newline comes up again. torch.multinomial(p, 1, generator=g) draws one id at random, each with its probability in p, and gives it back in a tensor; .item() turns it into a plain number. With p = torch.tensor([0.5, 0.0, 0.5]) you get 0 or 2, never 1.

The loss says how good the probabilities are: the average of minus the log of the probability given to each real next character in the file. Lower is better; a random guess among 19 characters scores log(19) = 2.944. When your code is done, it prints:

mimo
zasu
da
ni
nile
table: [19, 19]
k is followed by a 68 times and by e 64 times
after k: a 0.515 e 0.485 o 0.0
after a, the name ends with probability 0.393
every row adds up to 1: True
loss of the counting model: 1.496
loss of a random guess: 2.944

Your task: the starting code builds the vocabulary and encode from the last lessons, turns the whole file into ids, and has the loop that prints 5 names. Count the pairs of neighbours into counts, turn them into probs, and in the while loop draw each next id from the row of the current one.

Assignment
Follow the Coding Tutorial to build the table of counts, turn it into probabilities and sample 5 names from it.

Hint
The row is the character you are at, ids[i], and the column is the one after it, ids[i + 1]. In the while loop, the row to draw from is probs[ix].