Cross-Validation in Python

A test score from one split depends on which 75 customers happened to land in the test part. Change random_state and the same tree scores anywhere from 0.653 to 0.787. If you report just one of those numbers, you are reporting your luck, not your model.

Cross-validation tests on every row instead. cross_val_score(model, X, y, cv=5) (from sklearn.model_selection) cuts the rows into 5 parts, called folds, and trains the model 5 times. Each time it tests on a different fold and learns from the other four, so every customer is tested exactly once. It gives back the 5 test scores as a numpy array.

tree = DecisionTreeClassifier(max_depth=2, random_state=0)
three = cross_val_score(tree, X, y, cv=3)   # 3 folds: 3 scores
print(three.round(2))

An array of scores has its own helpers. scores.round(3) rounds every number in it. scores.mean() gives back their average, the number to report. scores.std() gives back their standard deviation, the spread: how far the scores usually are from their mean. A big spread warns that one split can't be trusted alone.

Below the six one-split lines the starting code prints, your script must add:

Five scores: [0.683 0.783 0.767 0.75  0.667]
Mean: 0.73
Spread (std): 0.046

Your task: the starting code makes a depth-5 tree in model and prints its test score on six different random splits. Below the loop, cross-validate the same model on all of X and y with 5 folds, then print the five scores, their mean and their spread.

Assignment
Follow the Coding Tutorial to cross-validate the tree and print its five scores, their mean and their spread.

Hint
Give cross_val_score all of X and y, not the split parts: it does the splitting itself. Round the array with scores.round(3) and the single numbers with round(..., 3).