Start learning free

LESSON 16 OF 165 · MACHINE LEARNING: PANDAS

Clean Messy Text in Python

Practice

Text typed by people is messy. In customers.csv one city is written several ways: "Madrid" and "madrid", "Cluj" and " cluj ", "Berlin" and "BERLIN". pandas compares text exactly, so it counts each spelling as a different city.

value_counts() gives back each different value of a column once, with how many rows have it, the biggest count first; it leaves out the empty cells. unique() gives back each different value once, empty included (as nan); wrapped in list(...), it prints every text in quotes, so stray spaces show.

A column holds many texts, so text methods go through .str, which runs them on every cell. str.strip() gives back each text without the spaces at its start and end, and str.title() gives back each word with one capital letter and the rest small. Empty cells stay empty; fillna("Unknown") gives them a name, so they get counted too.

names = pd.Series([" ana ", "BEN", "ana"])
names = names.str.strip().str.title()
print(names.value_counts().to_string())   # Ana 2, Ben 1

Your program must print:

city
Cluj      50
Lyon      48
Rome      47
Porto     46
Madrid    45
Berlin    43
madrid     6
Rome       3
 cluj      2
BERLIN     1
['Rome', 'Madrid', 'Lyon', 'Porto', 'Berlin', 'Cluj', 'madrid', ' cluj ', nan, 'Rome ', 'BERLIN']
city
Cluj       52
Madrid     51
Rome       50
Lyon       48
Porto      46
Berlin     44
Unknown     9

Your task: print the city counts and the list of different spellings, clean the city column with str.strip and str.title, fill the blank cities with "Unknown", and print the counts again.


Assignment
Follow the Coding Tutorial to clean the city column and count it before and after.


Hint
Each cleaning line stores its result back in df["city"]. .str.strip() and .str.title() can be chained on one line.


STEP 1 OF 5
    Ctrl + Enter to check

    Pd Clean Text in Python

    Keep going with Python1,600+ step-by-step lessons, from your first line of code to coding interviews. Free to start.
    Start learning free