Concepts 🚀 Project 🧮 Labs

Overfitting — the #1 reason ML models fail in production

Understand the bias-variance tradeoff, why train > test means overfitting, and 5 techniques to fix it. This is what separates seniors from freshers.

script.py v3.1

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

# ── Generate data with a TRUE relationship: y = sin(x) + noise ──
np.random.seed(42)
n = 30
x = np.sort(np.random.uniform(0, 5, n))
y_true = np.sin(x)
y = y_true + np.random.normal(0, 0.2, n)

# ── Train models with INCREASING complexity (polynomial degree) ──
degrees = [1, 3, 5, 15]
fig, axes = plt.subplots(1, 4, figsize=(20, 5))

for ax, degree in zip(axes, degrees):
    # Pipeline: polynomial features + linear regression
    model = Pipeline([
        ('poly', PolynomialFeatures(degree=degree)),
        ('linear', LinearRegression())
    ])
    model.fit(x.reshape(-1, 1), y)

    # Train + test scores
    x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=42)
    model.fit(x_train.reshape(-1, 1), y_train)
    train_score = model.score(x_train.reshape(-1, 1), y_train)
    test_score = model.score(x_test.reshape(-1, 1), y_test)

    # Plot
    x_plot = np.linspace(0, 5, 100)
    y_plot = model.predict(x_plot.reshape(-1, 1))
    ax.scatter(x_train, y_train, color='#a0c000', label='train', s=30)
    ax.scatter(x_test, y_test, color='#f85149', label='test', s=30, marker='x')
    ax.plot(x_plot, y_plot, color='#58a6ff', linewidth=2, label='model')
    ax.plot(x_plot, np.sin(x_plot), color='#888', linestyle='--', label='true (sin)')
    ax.set_title(f'Degree {degree}\ntrain R²={train_score:.3f}, test R²={test_score:.3f}')
    ax.set_ylim(-2, 2)
    ax.legend(fontsize=8)

plt.tight_layout()
plt.savefig('overfitting.png', dpi=100, bbox_inches='tight')
plt.show()

# ════════════════════════════════════════════════════════════════════
#  WHAT'S HAPPENING — bias vs variance
# ════════════════════════════════════════════════════════════════════
print('='*60)
print('BIAS-VARIANCE TRADEOFF')
print('='*60)
print('Degree 1 (UNDERFITTING): high bias, low variance')
print('  train R² low, test R² low. Model is too simple.')
print('  The model can\'t capture the sin curve — it\'s a straight line.')
print()
print('Degree 3 (GOOD FIT): balanced bias + variance')
print('  train R² high, test R² high. Model captures the pattern.')
print()
print('Degree 5 (SLIGHT OVERFIT): low bias, higher variance')
print('  train R² very high, test R² slightly lower. Starting to memorize.')
print()
print('Degree 15 (OVERFITTING): very low bias, very high variance')
print('  train R² = 1.0 (perfect), test R² NEGATIVE (worse than mean)!')
print('  Model memorized the training points + noise. Fails on new data.')

# ════════════════════════════════════════════════════════════════════
#  5 TECHNIQUES TO FIX OVERFITTING
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('5 TECHNIQUES TO FIX OVERFITTING')
print('='*60)
print('1. MORE DATA: the best fix. More data = model can\'t memorize all of it.')
print('2. SIMPLER MODEL: reduce polynomial degree, tree depth, neural net layers.')
print('3. REGULARIZATION: L1 (Lasso) / L2 (Ridge) — penalize large coefficients.')
print('4. CROSS-VALIDATION: use 5-fold CV to detect overfitting early.')
print('5. EARLY STOPPING: stop training when validation score starts dropping.')
print('   (Used in gradient boosting + neural nets.)')
print()
print('ALWAYS look at BOTH train + test scores:')
print('  train >> test = OVERFITTING (reduce complexity)')
print('  train ≈ test = good fit (or both bad = underfitting)')
print('  train low = UNDERFITTING (increase complexity)')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Overfitting = model memorizes training data instead of learning patterns. Symptom: train R² = 0.99, test R² = 0.50. The gap = overfitting. BIAS-VARIANCE TRADEOFF: High bias (underfitting): model too simple. train + test both low. High variance (overfitting): model too complex. train high, test low. Sweet spot: train ≈ test, both high. WHY IT MATTERS: production = new data = the test set scenario. A model that's 99% on train but 50% on test is WORTHLESS in production. 5 FIXES: (1) More data — best fix. Can't memorize 1M rows. (2) Simpler model — reduce degree/depth/layers. (3) Regularization — L1 (Lasso) zeros out useless features. L2 (Ridge) shrinks coefficients. (4) Cross-validation — 5-fold CV shows if the model generalizes. (5) Early stopping — stop when validation score drops. ALWAYS report train + test scores. If someone only shows train, assume they're hiding overfitting.

✏️ Try these changes

Change degree 15 to 25
→ even more extreme overfitting — test R² very negative
Increase n from 30 to 300
→ more data → less overfitting, even at degree 15
Change noise from 0.2 to 1.0
→ harder problem — all models do worse
Add Ridge regression to the pipeline: from sklearn.linear_model import Ridge
→ regularization reduces overfitting

← Previous lab

Model selection — which algorithm for which problem?

Next lab →

Full ML workflow — the senior data scientist's process