What Really Matters: Permutation Importance in Python

A trained tree model keeps built-in feature_importances_: how much each column's splits helped the trees fit the training rows. A column the trees used to fit quirks of those rows can look important there and still not help with new customers.

Permutation importance asks the test rows instead. Shuffle one column's values among the test customers, so each customer gets someone else's value, score the model again and see how far the score falls. A big drop means the model really needs that column. A drop near 0, or below it, means the model does as well without the column's real values.

permutation_importance, from sklearn.inspection, does this for every column. You give it a fitted model, rows and their labels; n_repeats says how many times each column is shuffled. It gives back a result whose importances_mean holds each column's average drop, in the order of the columns. It works with any model, even one with no built-in importances, like k-nearest neighbors:

from sklearn.neighbors import KNeighborsClassifier
from sklearn.inspection import permutation_importance

knn = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)
res = permutation_importance(knn, X_test, y_test, n_repeats=3, random_state=0)
print(res.importances_mean.round(3))

Your program must print:

Test rows: 0.747
Built-in importances, from the training rows:
visits             0.420
spent              0.193
days_since_last    0.189
orders             0.186
returned_item      0.012
Score drop when one column is shuffled, test rows:
visits             0.056
days_since_last    0.021
spent              0.011
orders             0.005
returned_item     -0.008

Your task: the starting code trains GradientBoostingClassifier(n_estimators=50, max_depth=2, random_state=0) and prints its test score and its built-in importances, sorted. Below the blank line, call permutation_importance with model, X_test, y_test, n_repeats=5 and random_state=0 and keep it in result. Put result.importances_mean in a pandas Series drops with index=X.columns, print the title Score drop when one column is shuffled, test rows:, then print drops from biggest to smallest, rounded to 3 digits.

Assignment
Follow the Coding Tutorial to measure how far the test score falls when each column is shuffled, and print the drops sorted.

Hint
Print drops the same way the starting code prints built_in: sort_values(ascending=False), then .round(3), then .to_string().