Cross-validation — more honest than a single split

Single train/test split = lucky or unlucky. 5-fold CV runs 5 different splits and averages — more honest.

script.py v3.1

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import cross_validate, KFold

X, y = make_regression(n_samples=500, n_features=5, noise=20, random_state=42)

# ── KFold: defines HOW to split ──
cv = KFold(n_splits=5, shuffle=True, random_state=42)
print(f'5-fold CV: data split into 5 pieces, train on 4, test on 1, repeat 5x')

# ── cross_validate: runs the CV + computes metrics ──
# scoring uses sklearn's internal names. 'neg_' prefix because sklearn MAXIMIZES scores.
scoring = {
    'MAE': 'neg_mean_absolute_error',
    'MSE': 'neg_mean_squared_error',
    'R2': 'r2',
}

def benchmark_cv(model, X, y, name):
    results = cross_validate(model, X, y, cv=cv, scoring=scoring, return_train_score=False)

    # results['test_MAE'] is an array of 5 values (one per fold)
    # Negate because sklearn negated them (it maximizes)
    mae_scores = -results['test_MAE']
    mse_scores = -results['test_MSE']
    r2_scores = results['test_R2']

    print(f'\n{name}:')
    print(f'  MAE per fold:  {mae_scores.round(2)}')
    print(f'  MAE mean ± std: {mae_scores.mean():.2f} ± {mae_scores.std():.2f}')
    print(f'  MSE mean:       {mse_scores.mean():.2f}')
    print(f'  RMSE mean:      {np.sqrt(mse_scores.mean()):.2f}')
    print(f'  R² per fold:    {r2_scores.round(4)}')
    print(f'  R² mean ± std:  {r2_scores.mean():.4f} ± {r2_scores.std():.4f}')
    return r2_scores.mean()

print('Cross-validation benchmark:')
print('=' * 60)
lr_r2 = benchmark_cv(LinearRegression(), X, y, 'LinearRegression')
rf_r2 = benchmark_cv(RandomForestRegressor(n_estimators=50, random_state=42), X, y, 'RandomForest')

print(f'\nWinner (higher R²): {"RandomForest" if rf_r2 > lr_r2 else "LinearRegression"}')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Why CV? A single train/test split is one sample from many possible splits. You might get lucky (easy test set) or unlucky (hard test set). 5-fold CV runs 5 different splits and averages — more honest estimate. KFold(n_splits=5, shuffle=True) — split into 5 pieces, shuffle first. cross_validate(model, X, y, cv=cv, scoring=...) — runs the CV, returns a dict of arrays. Each metric is an array of 5 values (one per fold). Report MEAN ± STD. Std tells you how stable the model is — high std = sensitive to the data split. neg_ prefix: sklearn maximizes scores, so errors are negated (maximizing -MAE = minimizing MAE).

✏️ Try these changes

Change n_splits from 5 to 10
→ 10-fold CV — more folds, slower, slightly more accurate
Change shuffle from True to False
→ no shuffling — if data is sorted, CV is biased
Change n_estimators from 50 to 200
→ RF R² might improve
Add return_train_score=True and compare train vs test R²
→ if train >> test, model is overfitting

← Previous lab

Classification metrics — accuracy, precision, recall, F1

Next lab →

Handle ANY model type (sklearn, xgboost, custom)