How Many Trees, How Big a Step in Python

Boosting has two settings that work together. n_estimators is how many trees it adds. learning_rate is how much of each tree's correction really counts: with 1.0 every tree takes its full step, with 0.1 only a tenth of it. Big steps learn fast but soon start learning the noise in the training rows; small steps need more trees to get anywhere.

To see this you need the model's answers after 5 trees, after 20 and so on, without training it again each time. model.staged_predict(X) does that: it gives back the predictions for X again and again, first from tree 1 alone, then from trees 1 and 2, and so on up to all of them. accuracy_score(y_true, y_pred) from sklearn.metrics gives back the share of the predictions that are right. On a small model with 3 trees:

small = GradientBoostingClassifier(n_estimators=3, random_state=0)
small.fit(X_train, y_train)
for answers in small.staged_predict(X_test.head(4)):
    print(answers)   # 3 lines: 4 guesses after 1, 2 and 3 trees

The starting code trains the same model twice, with 200 trees of depth 2 and learning_rate 1.0, then 0.1, and prints its share right on the training rows. You add a loop that prints its test accuracy after 5, 20, 50 and 200 trees. Your script must print:

learning_rate 1.0 | training rows after 200 trees: 1.0
  after 5 trees, test rows: 0.76
  after 20 trees, test rows: 0.72
  after 50 trees, test rows: 0.747
  after 200 trees, test rows: 0.733
learning_rate 0.1 | training rows after 200 trees: 0.96
  after 5 trees, test rows: 0.627
  after 20 trees, test rows: 0.773
  after 50 trees, test rows: 0.747
  after 200 trees, test rows: 0.747

With big steps the best of these lines comes after 5 trees; then the test score slips while the training score climbs to 1.0, because the extra trees memorize the training rows. Small steps start slow and reach the best line of the run, 0.773, after 20 trees.

Your task: inside the for rate loop, below the print, set a counter trees to 0 and loop over model.staged_predict(X_test), calling each item predictions. Add 1 to trees each time, and when trees is 5, 20, 50 or 200, print " after", trees, "trees, test rows:" and the accuracy_score of predictions, rounded to 3 digits.

Assignment
Follow the Coding Tutorial to print the test accuracy after 5, 20, 50 and 200 trees for both learning rates.

Hint
The counter goes back to 0 once per learning rate, before the inner loop starts. Inside the inner loop, add 1 first, then check it with in against the list [5, 20, 50, 200].