Concepts 🚀 Project 🧮 Labs

Gradient Boosting visualized — sequential learning + residuals (breast cancer)

SEE how each tree corrects the PREVIOUS tree's errors. SEE how learning rate + n_estimators interact. The competition-winning algorithm.

script.py v3.1

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# ── Load breast cancer (30 features, binary: malignant vs benign) ──
cancer = load_breast_cancer()
X = cancer.data
y = cancer.target  # 0=malignant, 1=benign

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

print(f'Dataset: breast cancer ({len(y)} samples, {X.shape[1]} features)')
print(f'Classes: {cancer.target_names} (0=malignant, 1=benign)')

# ════════════════════════════════════════════════════════════════════
#  PLOT 1: Test accuracy vs n_estimators (gradient boosting vs random forest)
# ════════════════════════════════════════════════════════════════════
fig, axes = plt.subplots(1, 3, figsize=(18, 5))

n_est_range = [1, 5, 10, 25, 50, 100, 200]
gb_train = []
gb_test = []
rf_train = []
rf_test = []

for n in n_est_range:
    gb = GradientBoostingClassifier(n_estimators=n, learning_rate=0.1, random_state=42)
    gb.fit(X_train, y_train)
    gb_train.append(gb.score(X_train, y_train))
    gb_test.append(gb.score(X_test, y_test))

    rf = RandomForestClassifier(n_estimators=n, random_state=42)
    rf.fit(X_train, y_train)
    rf_train.append(rf.score(X_train, y_train))
    rf_test.append(rf.score(X_test, y_test))

axes[0].plot(n_est_range, gb_test, 'o-', color='#a0c000', label='GradientBoosting (test)')
axes[0].plot(n_est_range, rf_test, 's-', color='#58a6ff', label='RandomForest (test)')
axes[0].set_xlabel('n_estimators')
axes[0].set_ylabel('Test Accuracy')
axes[0].set_title('GradientBoosting vs RandomForest')
axes[0].legend()
axes[0].set_xscale('log')
axes[0].grid(True, alpha=0.3)

# ════════════════════════════════════════════════════════════════════
#  PLOT 2: Effect of learning_rate (the most important parameter)
# ════════════════════════════════════════════════════════════════════
lr_values = [0.01, 0.05, 0.1, 0.3, 1.0]
for lr in lr_values:
    gb = GradientBoostingClassifier(n_estimators=100, learning_rate=lr, random_state=42)
    gb.fit(X_train, y_train)
    train_scores = list(gb.staged_score(X_train, y_train))
    test_scores = list(gb.staged_score(X_test, y_test))
    axes[1].plot(range(1, 101), test_scores, label=f'lr={lr}')

axes[1].set_xlabel('n_estimators (iterations)')
axes[1].set_ylabel('Test Accuracy')
axes[1].set_title('Learning rate effect\n(low lr = slow but stable, high lr = fast but might overshoot)')
axes[1].legend()
axes[1].grid(True, alpha=0.3)

# ════════════════════════════════════════════════════════════════════
#  PLOT 3: Feature importance (which features drive the prediction?)
# ════════════════════════════════════════════════════════════════════
gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
gb.fit(X_train, y_train)

importances = gb.feature_importances_
indices = np.argsort(importances)[-10:]  # top 10
axes[2].barh(range(10), importances[indices], color='#bc8cff', edgecolor='black')
axes[2].set_yticks(range(10))
axes[2].set_yticklabels([cancer.feature_names[i] for i in indices], fontsize=8)
axes[2].set_xlabel('Importance')
axes[2].set_title('Top 10 features\n(breast cancer)')

plt.tight_layout()

print(f'\nGradientBoosting (100 trees, lr=0.1):')
print(f'  Train accuracy: {gb.score(X_train, y_train):.4f}')
print(f'  Test accuracy:  {gb.score(X_test, y_test):.4f}')
print(f'\nTop 5 important features:')
for i in np.argsort(importances)[-5:][::-1]:
    print(f'  {cancer.feature_names[i]:30s}  {importances[i]:.4f}')

print(f'\nTHE MATHS (Gradient Boosting):')
print(f'  Unlike RandomForest (trees built INDEPENDENTLY),')
print(f'  Gradient Boosting builds trees SEQUENTIALLY — each fixes the PREVIOUS tree\'s errors.')
print(f'\n  Algorithm:')
print(f'    1. Start with a simple prediction (the mean): F₀(x) = mean(y)')
print(f'    2. Compute RESIDUALS: rᵢ = yᵢ - F₀(xᵢ)')
print(f'    3. Train tree h₁ to predict the RESIDUALS (not y!)')
print(f'    4. Update: F₁(x) = F₀(x) + lr * h₁(x)')
print(f'    5. New residuals: rᵢ = yᵢ - F₁(xᵢ)')
print(f'    6. Train tree h₂ on new residuals. Repeat.')
print(f'\n  Each tree learns from the MISTAKES of the ensemble so far.')
print(f'  The LEARNING RATE (lr) controls how much each tree contributes:')
print(f'    lr=0.01: slow learning, needs many trees, very stable')
print(f'    lr=0.1:  good default')
print(f'    lr=1.0:  fast but might overshoot (like GD with high lr)')
print(f'\n  lr × n_estimators tradeoff:')
print(f'    Low lr + many trees = best accuracy (but slower training)')
print(f'    High lr + few trees = fast but lower accuracy')
print(f'    Rule: lr=0.1, n_estimators=100-1000 is a good starting point')

print(f'\n  vs RandomForest:')
print(f'    RF: trees are INDEPENDENT (bagging). Reduces variance.')
print(f'    GB: trees are SEQUENTIAL (boosting). Reduces bias.')
print(f'    GB usually wins on tabular data (Kaggle competitions).')
print(f'    RF is simpler + faster (trees can train in parallel).')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Gradient Boosting — VISUALIZED: THE KEY DIFFERENCE from RandomForest: RF builds trees INDEPENDENTLY (bagging) → reduces VARIANCE. GB builds trees SEQUENTIALLY (boosting) → reduces BIAS. ALGORITHM: (1) Start with mean prediction: F₀(x) = mean(y). (2) Compute RESIDUALS: r = y - F₀(x). (How wrong is the current model?) (3) Train a tree to predict the RESIDUALS (not y). (4) Update: F₁ = F₀ + lr × tree₁. (Add the correction.) (5) New residuals → train tree₂ → update → repeat. EACH TREE FIXES THE PREVIOUS ERRORS: Tree 1 captures the main pattern. Tree 2 captures what tree 1 got wrong. Tree 3 captures what trees 1+2 got wrong. ... like a team of specialists, each fixing what the others miss. LEARNING RATE (lr): how much each tree contributes. lr=0.01: slow learning. Needs many trees (500+). Very stable. lr=0.1: good default. 100 trees usually enough. lr=1.0: fast but might overshoot (like GD with high lr). RULE: lr × n_estimators should be roughly constant. lr=0.1, n=100 ≈ lr=0.01, n=1000 (same complexity, latter is slightly better). WHY GB WINS KAGGLE: On tabular data, GB (XGBoost/LightGBM) beats deep learning ~80% of the time. Handles missing values, categorical features, non-linear interactions. The staged_score plot shows accuracy at each iteration — you can see exactly when adding more trees stops helping (early stopping). BREAST CANCER: 'worst concave points' + 'worst area' are usually top features. These are medical measurements — GB finds the most predictive ones automatically.

✏️ Try these changes

Change learning_rate from 0.1 to 0.01
→ slower learning — needs more trees for same accuracy
Change learning_rate from 0.1 to 1.0
→ fast but unstable — test accuracy might drop
Add max_depth=3 to the model
→ deeper trees — more complex, might overfit
Add subsample=0.8
→ stochastic gradient boosting — each tree sees 80% of data, reduces overfitting

← Previous lab

K-Means visualized — centroid movement + elbow plot (iris)

Next lab →

Interactive: Linear Regression — drag the slope + intercept