Start learning free

LESSON 123 OF 129 · AI ENGINEERING: INTERVIEW QUESTIONS

Score an Answer in Python

Practice

Interview question: “Your bot answers questions about the shop, and for each test question you have the right answer, the truth. How do you score the bot's answers automatically?” The two classic metrics are exact match and token F1.

Both start by normalizing the two strings: lower case, no punctuation, one space between words. Exact match then asks whether the two are the same string, True or False. It is strict: a right answer with a few extra words gets nothing.

Token F1 gives partial credit for the words the two share. Precision is the shared words divided by the answer's words (how much of the answer is right). Recall is the shared words divided by the truth's words (how much of the truth the answer holds). F1 combines them: 2 × precision × recall / (precision + recall). For example:

truth:   Ships in 2 days.     →  ships in 2 days      (4 words)
answer:  It ships in 3 days   →  it ships in 3 days   (5 words)
shared:  ships, in, days      →  3 words
precision = 3 / 5 = 0.6
recall    = 3 / 4 = 0.75
F1 = 2 × 0.6 × 0.75 / (0.6 + 0.75) = 0.667

When nothing is shared, F1 is 0, and you must return it before dividing: precision + recall would be 0, and Python can't divide by 0.

The starting code has the truth, five answers, a normalize that doesn't remove punctuation yet, and count_shared, which matches each truth word at most once. Finished, the script prints each answer in brackets, its exact match and its F1 rounded to 3 digits:

[within 14 days] True 1.0
[  Within 14   DAYS!] True 1.0
[You can return it within 14 days.] False 0.6
[30 days] False 0.4
[No idea.] False 0.0

Your task: make normalize remove every mark in string.punctuation, set exact in the bottom loop, and make f1 return the F1 of precision and recall, or 0.0 when no words are shared.


Assignment
Follow the Coding Tutorial to finish normalize, add exact match and build token F1, with its check for no shared words.


Hint
Removing a mark is a replace with an empty string. Put brackets around precision + recall, and check shared == 0 before computing precision: No idea. shares no words with the truth.


STEP 1 OF 8
    Ctrl + Enter to check

    Aiq Exact Match F1 in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free