Random Forest visualized — many trees + feature importance (wine)

SEE how 100 trees vote. SEE feature importance. SEE OOB error stabilize. The wine dataset (13 features, 3 classes).

script.py v3.1

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# ── Load wine dataset (13 chemical features, 3 wine cultivars) ──
wine = load_wine()
X = wine.data
y = wine.target
feature_names = wine.feature_names

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

print(f'Dataset: wine ({len(y)} samples, {X.shape[1]} features, 3 classes)')
print(f'Features: {feature_names}')

# ════════════════════════════════════════════════════════════════════
#  PLOT 1: Accuracy vs n_estimators — how many trees do you need?
# ════════════════════════════════════════════════════════════════════
n_trees_list = [1, 5, 10, 20, 50, 100, 200]
train_scores = []
test_scores = []
oob_scores = []

for n in n_trees_list:
    rf = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=42)
    rf.fit(X_train, y_train)
    train_scores.append(rf.score(X_train, y_train))
    test_scores.append(rf.score(X_test, y_test))
    oob_scores.append(rf.oob_score_ if n > 1 else np.nan)

fig, axes = plt.subplots(1, 3, figsize=(18, 5))

axes[0].plot(n_trees_list, train_scores, 'o-', color='#a0c000', label='train')
axes[0].plot(n_trees_list, test_scores, 's-', color='#58a6ff', label='test')
axes[0].plot(n_trees_list, oob_scores, '^-', color='#f85149', label='OOB (out-of-bag)')
axes[0].set_xlabel('Number of trees')
axes[0].set_ylabel('Accuracy')
axes[0].set_title('How many trees do you need?')
axes[0].legend()
axes[0].set_xscale('log')
axes[0].axhline(1.0, color='gray', linestyle='--', alpha=0.3)

# ════════════════════════════════════════════════════════════════════
#  PLOT 2: Feature importance — which features matter most?
# ════════════════════════════════════════════════════════════════════
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

importances = rf.feature_importances_
indices = np.argsort(importances)[::-1]  # sort descending

axes[1].barh(range(len(importances)), importances[indices][::-1],
            color='#bc8cff', edgecolor='black')
axes[1].set_yticks(range(len(importances)))
axes[1].set_yticklabels([feature_names[i] for i in indices][::-1], fontsize=8)
axes[1].set_xlabel('Importance (0-1)')
axes[1].set_title('Feature Importance\n(which features matter most?)')
for i, v in enumerate(importances[indices][::-1]):
    axes[1].text(v + 0.005, i, f'{v:.3f}', va='center', fontsize=7)

# ════════════════════════════════════════════════════════════════════
#  PLOT 3: Individual tree predictions vs forest vote (on 1 sample)
# ════════════════════════════════════════════════════════════════════
# Take a test sample and see how each tree votes
sample_idx = 0
sample = X_test[sample_idx:sample_idx+1]
true_label = y_test[sample_idx]

tree_votes = []
for tree in rf.estimators_:
    tree_votes.append(tree.predict(sample)[0])

class_counts = [tree_votes.count(c) for c in range(3)]
colors = ['#e74c3c', '#2ecc71', '#3498db']
axes[2].bar(['Class 0\n(' + wine.target_names[0] + ')',
             'Class 1\n(' + wine.target_names[1] + ')',
             'Class 2\n(' + wine.target_names[2] + ')'],
            class_counts, color=colors, edgecolor='black')
axes[2].set_ylabel('Number of trees voting for this class')
axes[2].set_title(f'Forest vote on 1 sample\n(true: {wine.target_names[true_label]}, '
                  f'predicted: {wine.target_names[rf.predict(sample)[0]]})')
for i, count in enumerate(class_counts):
    axes[2].text(i, count + 0.5, str(count), ha='center', fontsize=12, fontweight='bold')

plt.tight_layout()

print(f'\nResults (100 trees):')
print(f'  Train accuracy: {rf.score(X_train, y_train):.3f}')
print(f'  Test accuracy:  {rf.score(X_test, y_test):.3f}')
print(f'  OOB score:      {rf.oob_score_:.3f}')
print(f'\nTop 5 important features:')
for i in indices[:5]:
    print(f'  {feature_names[i]:30s}  {importances[i]:.4f}')
print(f'\nTHE MATHS:')
print(f'  Random Forest = ensemble of decision trees + voting.')
print(f'  Each tree sees a RANDOM SUBSET of rows (bootstrap = sample with replacement).')
print(f'  At each split, only a RANDOM SUBSET of features is considered.')
print(f'  Final prediction = MAJORITY VOTE (classification) or AVERAGE (regression).')
print(f'\n  Why does this work?')
print(f'    Single tree: high variance (small data change → different tree)')
print(f'    Many trees:  variance averages out → low variance + low bias')
print(f'    This is BAGGING (Bootstrap Aggregating).')
print(f'\n  OOB (Out-Of-Bag) score:')
print(f'    Each tree trains on ~63% of data (bootstrap). The other ~37% is "out-of-bag".')
print(f'    OOB score = accuracy on the OOB samples. FREE validation — no separate test set needed.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Random Forest — VISUALIZED: WHAT: 100+ decision trees + majority vote. The wisdom of the crowd. HOW IT WORKS: (1) BOOTSTRAP: each tree gets a random sample of rows (with replacement). ~63% unique rows per tree. (2) RANDOM FEATURES: at each split, only √n_features are considered. Prevents all trees from picking the same split. (3) VOTE: final prediction = majority vote (classification) or average (regression). WHY IT WORKS (BAGGING): Single tree: high variance (change 1 row → different tree). Many trees: variance averages out. Same bias, lower variance. Mathematically: Var(average of n) = Var(single) / n. More trees = less variance. OOB SCORE (free validation): Each tree trains on ~63% of data. The other ~37% (out-of-bag) is unseen. OOB score = accuracy on OOB samples. No need for a separate validation set. FEATURE IMPORTANCE: Measured by how much each feature REDUCES gini impurity across ALL trees. Summed + averaged. Higher = more important. On wine: 'flavanoids' and 'color_intensity' are usually top — they're chemically meaningful. HOW MANY TREES: 1 tree = high variance. 10 trees = good. 100 = stable. 200+ = no improvement. The curve PLATEAUS — more trees don't help after ~100. Just wastes memory. NEVER increases overfitting (unlike max_depth). More trees = always better or same.

✏️ Try these changes

Change n_estimators from 100 to 5
→ fewer trees → higher variance → test accuracy drops slightly
Change max_features from default (sqrt) to 1.0 (all features)
→ trees are more correlated → bagging helps less
Set oob_score=False
→ can't track OOB — need a separate validation set
Use load_breast_cancer instead of load_wine
→ 30 features → different importance ranking

← Previous lab

Decision Tree visualized — the splits + overfitting (iris)

Next lab →

KNN visualized — the K nearest neighbors + decision boundary (iris)