Concepts 🚀 Project 🧮 Labs

Gradient Descent — the engine behind neural networks

Most ML models (neural nets, logistic regression, SVMs) use gradient descent, not closed-form solutions. Understand the algorithm.

script.py v3.1

import numpy as np
import matplotlib.pyplot as plt

# ── Gradient Descent: find the minimum of a function by walking downhill ──
# f(x) = x² — minimum at x=0
# f'(x) = 2x — gradient points UPHILL. We go in -gradient direction.

# ── Visualize the function ──
x = np.linspace(-5, 5, 100)
y = x**2
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].plot(x, y, color='#a0c000', linewidth=2)
axes[0].set_xlabel('x'); axes[0].set_ylabel('f(x) = x²')
axes[0].set_title('Function to minimize')

# ── Run gradient descent ──
def gradient_descent(start_x, learning_rate, n_steps):
    """Minimize f(x) = x² using gradient descent."""
    x = start_x
    history = [x]
    for step in range(n_steps):
        # Gradient of x² is 2x
        gradient = 2 * x
        # Update: x = x - learning_rate * gradient
        x = x - learning_rate * gradient
        history.append(x)
    return history

# Run with different learning rates
lr_small = gradient_descent(start_x=4, learning_rate=0.1, n_steps=20)
lr_good = gradient_descent(start_x=4, learning_rate=0.5, n_steps=20)
lr_large = gradient_descent(start_x=4, learning_rate=1.1, n_steps=20)

axes[1].plot(lr_small, [x**2 for x in lr_small], 'o-', color='#58a6ff', label='lr=0.1 (slow)')
axes[1].plot(lr_good, [x**2 for x in lr_good], 'o-', color='#a0c000', label='lr=0.5 (good)')
axes[1].plot(lr_large, [x**2 for x in lr_large], 'o-', color='#f85149', label='lr=1.1 (DIVERGES)')
axes[1].set_xlabel('Step'); axes[1].set_ylabel('f(x)')
axes[1].set_title('Gradient Descent — learning rate matters!')
axes[1].legend()
plt.tight_layout()
plt.savefig('gradient_descent.png', dpi=100, bbox_inches='tight')
plt.show()

print('Gradient Descent Results:')
print(f'  lr=0.1 (slow):  final x={lr_small[-1]:.4f} (took 20 steps, still not at 0)')
print(f'  lr=0.5 (good):  final x={lr_good[-1]:.4f} (reached 0 fast)')
print(f'  lr=1.1 (large): final x={lr_large[-1]:.4f} (DIVERGED — bouncing away!)')

# ════════════════════════════════════════════════════════════════════
#  THE ALGORITHM — step by step
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('THE ALGORITHM')
print('='*60)
print('1. Initialize parameters randomly: β = random')
print('2. Compute gradient: ∇f = ∂f/∂β (points uphill)')
print('3. Update: β = β - learning_rate * ∇f (step downhill)')
print('4. Repeat until convergence (gradient ≈ 0)')
print()
print('LEARNING RATE (lr):')
print('  Too small: slow convergence (1000s of steps)')
print('  Too large: DIVERGES (bounces away from minimum)')
print('  Just right: fast convergence (10-100 steps)')
print('  Typical values: 0.001, 0.01, 0.1')

# ════════════════════════════════════════════════════════════════════
#  VARIANTS — SGD, Mini-batch, Adam
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('VARIANTS')
print('='*60)
print('BATCH GD: use ALL data per step. Accurate gradient, slow per step.')
print('STOCHASTIC GD (SGD): use 1 sample per step. Noisy gradient, fast per step.')
print('MINI-BATCH GD: use 32-256 samples. Best of both — standard for neural nets.')
print('ADAM: adaptive learning rate per parameter. Most popular optimizer.')
print('  Combines momentum (smooths gradient) + RMSprop (adapts lr per param).')
print('  Default for deep learning. Works well with lr=0.001.')

# ════════════════════════════════════════════════════════════════════
#  WHY IT MATTERS — which ML models use GD?
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('WHICH MODELS USE GRADIENT DESCENT?')
print('='*60)
print('LinearRegression: NO — uses closed-form (Normal Equation)')
print('LogisticRegression: YES — no closed-form solution')
print('Neural Networks: YES — ALWAYS (backpropagation = chain rule + GD)')
print('SVM: YES — uses SMO (a variant of GD)')
print('XGBoost/LightGBM: YES — gradient boosting = GD on residuals')
print('RandomForest: NO — uses greedy tree building')
print()
print('If you understand GD, you understand 80% of ML training.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Gradient Descent = the engine of modern ML. ALGORITHM: (1) init β random, (2) compute gradient ∇f (uphill direction), (3) step β -= lr * ∇f, (4) repeat. LEARNING RATE: too small = slow. Too large = diverge. Just right = fast. Start with 0.001, tune by 3x (0.001 → 0.003 → 0.01). VARIANTS: Batch GD — all data per step. Accurate, slow. SGD — 1 sample per step. Noisy, fast. Mini-batch (32-256) — standard. Best of both. Adam — adaptive lr per param + momentum. Default for deep learning. WHY IT MATTERS: neural nets, logistic regression, SVMs, gradient boosting — ALL use GD. LinearRegression + RandomForest are exceptions (closed-form / greedy). Backpropagation (neural nets) = chain rule of calculus + GD. If you understand GD, you understand how most models LEARN.

✏️ Try these changes

Change learning_rate from 0.5 to 0.01
→ much slower convergence — needs 200+ steps
Change learning_rate from 1.1 to 2.0
→ diverges faster — bounces to infinity
Change the function from x² to x⁴ (gradient = 4x³)
→ steeper — needs smaller lr
Add momentum: v = 0.9*v + 0.1*gradient; x -= lr*v
→ smoother convergence — Adam-like

← Previous lab

Linear Regression maths — the equation behind .fit()

Next lab →

Probability for ML — Bayes, distributions, maximum likelihood