Concepts 🚀 Project 🧮 Labs

Model selection — which algorithm for which problem?

Not 'try RandomForest'. A decision framework: data size, feature type, interpretability need, linearity assumption. Pick the RIGHT model.

script.py v3.1

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier, GradientBoostingRegressor
from sklearn.svm import SVR, SVC
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_regression, make_classification
from sklearn.model_selection import cross_val_score
import time

# ════════════════════════════════════════════════════════════════════
#  THE DECISION FRAMEWORK — 5 questions to pick a model
# ════════════════════════════════════════════════════════════════════
print('='*60)
print('MODEL SELECTION — 5 questions')
print('='*60)
print('Q1: REGRESSION or CLASSIFICATION?')
print('  Regression: target is continuous (price, age, temperature)')
print('  Classification: target is categorical (spam/not, species A/B/C)')
print()
print('Q2: How MUCH data?')
print('  < 100 rows: LinearRegression/LogisticRegression (low variance)')
print('  100-10k: anything works')
print('  > 100k: avoid KNN/SVM (slow), use trees or linear')
print()
print('Q3: LINEAR or NON-LINEAR relationship?')
print('  Linear (scatter plot shows a line): LinearRegression, LogisticRegression')
print('  Non-linear (curve, clusters): RandomForest, GradientBoosting, SVM(rbf)')
print()
print('Q4: Need INTERPRETABILITY?')
print('  Yes (medical, legal, finance): LinearRegression, DecisionTree, LogisticRegression')
print('  No (competitions, raw accuracy): RandomForest, XGBoost, Neural Net')
print()
print('Q5: How many FEATURES?')
print('  < 50: anything works')
print('  50-1000: LinearRegression, RandomForest, XGBoost')
print('  > 1000 (text, images): LinearModel (sparse), Neural Net, or reduce dimensions first')

# ════════════════════════════════════════════════════════════════════
#  BENCHMARK: compare 6 models on the same dataset
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('BENCHMARK: 6 regression models on the same data')
print('='*60)

X, y = make_regression(n_samples=500, n_features=10, noise=15, random_state=42)

models = {
    'LinearRegression':      LinearRegression(),
    'Ridge':                 Ridge(alpha=1.0),
    'DecisionTree':          DecisionTreeRegressor(max_depth=5, random_state=42),
    'RandomForest':          RandomForestRegressor(n_estimators=50, random_state=42),
    'GradientBoosting':      GradientBoostingRegressor(n_estimators=50, random_state=42),
    'KNN':                   KNeighborsRegressor(n_neighbors=5),
    'SVR':                   SVR(kernel='rbf'),
}

results = []
for name, model in models.items():
    t0 = time.perf_counter()
    scores = cross_val_score(model, X, y, cv=5, scoring='r2')
    elapsed = time.perf_counter() - t0
    results.append({
        'model': name,
        'R2_mean': scores.mean(),
        'R2_std': scores.std(),
        'time_sec': elapsed,
    })

df = pd.DataFrame(results).set_index('model')
print(df.round(3).to_string())

# ════════════════════════════════════════════════════════════════════
#  ANALYSIS — when to pick which
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('WHEN TO PICK WHICH')
print('='*60)
best_r2 = df['R2_mean'].idxmax()
fastest = df['time_sec'].idxmin()
print(f'Best R²:  {best_r2} ({df.loc[best_r2, "R2_mean"]:.4f})')
print(f'Fastest:  {fastest} ({df.loc[fastest, "time_sec"]:.3f}s)')
print()
print('LinearRegression: FAST, interpretable, baseline. Use FIRST.')
print('  Good when: linear relationship, few features, need interpretability.')
print('  Bad when: non-linear, high dimensions, multicollinearity (use Ridge).')
print()
print('RandomForest: accurate, handles non-linearity, no scaling needed.')
print('  Good when: non-linear, mixed feature types, medium data.')
print('  Bad when: > 1M rows (slow), need extrapolation (trees can\'t).')
print()
print('GradientBoosting (XGBoost/LightGBM): HIGHEST accuracy, competition winner.')
print('  Good when: max accuracy, tabular data, have time to tune.')
print('  Bad when: small data (overfits), need interpretability.')
print()
print('KNN: simple, no training. Just store data + vote.')
print('  Good when: very small data, few features, non-linear.')
print('  Bad when: many features (curse of dimensionality), large data (slow predict).')
print()
print('SVM: powerful with kernels, good for small data.')
print('  Good when: small data, high dimensions (text), clear margin.')
print('  Bad when: large data (O(n²) or worse), need probabilities.')
print()
print('NEURAL NETS: not in this benchmark. Use for: images, text, audio.')
print('  Bad for: tabular data (XGBoost usually wins), small data (overfits).')
print('  Rule: try XGBoost first. If it doesn\'t win, then try deep learning.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Model selection = matching algorithm to problem. 5 questions: Q1: Regression (continuous target) vs Classification (categorical target). Q2: Data size. <100 = linear models. 100-10k = anything. >100k = avoid KNN/SVM. Q3: Linear vs non-linear. Plot scatter. Linear line → LinearRegression. Curve → trees/SVM. Q4: Interpretability. Medical/legal/finance → linear/tree. Competition → XGBoost/neural net. Q5: Feature count. <50 = anything. 50-1000 = linear/trees. >1000 = linear/neural/reduce dims. RULE OF THUMB: Start with LinearRegression/LogisticRegression (baseline). Try RandomForest (handles non-linearity, no scaling). If accuracy matters most: XGBoost/LightGBM. If data is images/text/audio: neural nets. If data is small + high-dim: SVM. If you need interpretability: DecisionTree or linear. NEVER start with neural nets on tabular data — XGBoost almost always wins.

✏️ Try these changes

Change n_samples from 500 to 50
→ linear models will win — small data favors low-variance models
Change n_features from 10 to 100
→ KNN will do worse — curse of dimensionality
Change noise from 15 to 50
→ all models do worse — harder problem
Add make_classification + compare LogisticRegression vs RandomForestClassifier
→ classification version of the same benchmark

← Previous lab

Probability for ML — Bayes, distributions, maximum likelihood

Next lab →

Overfitting — the #1 reason ML models fail in production