Leakage: Answers Hiding in the Features in Python

Leakage is when a feature is written down after the thing you want to predict. The model finds the answer in its own inputs and scores almost perfectly on the test rows. But on the day you really predict, that column isn't filled in yet, so the great score never shows up.

In this lesson the shop's table has an extra column, thank_you_email_sent: the shop sends that email right after a customer buys again. The starting code gives you test_score(names): it splits the rows, trains a tree on the columns you name and gives back its score on the test rows, rounded to 3 digits.

Spotting it: on a DataFrame, .corr() gives back a table with one number for each pair of columns, from -1 to 1: how strongly they move together. Near 0 means no link; near 1 (or -1) means one column almost tells you the other. .round(3) gives back the numbers rounded, and .to_string() turns them into text with every row. To drop a name from a list, a list's .remove(item) takes that item out:

orders = pd.read_csv("orders.csv")
print(orders[["quantity", "unit_price"]].corr().round(2).to_string())

names = ["quantity", "unit_price", "status"]
names.remove("status")    # names is now ["quantity", "unit_price"]

Real habits move with the label a little. A column at almost 1 is the label in disguise: drop it, and the score falls to what the model can really do. Your script must print:

Every column: 0.987
visits                  0.496
orders                  0.488
spent                   0.402
days_since_last        -0.418
returned_item          -0.082
thank_you_email_sent    0.993
bought_again            1.000
Without it: 0.72

Your task: below the function, print the score with every column, keep the correlations of the columns plus "bought_again" in corr, print its "bought_again" column rounded to 3 digits, take "thank_you_email_sent" out of columns, and print the score again.

Assignment
Follow the Coding Tutorial to measure the leaky score, find the leak with .corr(), drop it and measure the honest score.

Hint
The column near 1.000 that isn't bought_again itself is the leak. columns + ["bought_again"] is a new list with the label added at the end, and corr["bought_again"] is the column of corr you print.