KNN visualized — the K nearest neighbors + decision boundary (iris)

KNN (K-Nearest Neighbors) — VISUALIZED: THE SIMPLEST ML ALGORITHM: no training. Just store the data. PREDICTION: find the K nearest training points, vote. DISTANCE: Euclidean d = √(Σ(xᵢ-yᵢ)²). MUST scale features first! Without scaling: a feature in thousands (salary) dominates a feature in single digits (age). K CONTROLS SMOOTHNESS: K=1 (OVERFIT): each point is its own neighbor → 100% train accuracy. Boundary is JAGGED — every training point creates its own island. Captures NOISE as if it were signal. K=5 (GOOD): smooth boundary, captures the real pattern. K=15 (SMOOTH): very smooth, might miss fine details. K=50 (UNDERFIT): too smooth — predicts the majority class everywhere. K vs BIAS-VARIANCE: Low K = low bias (flexible) + high variance (sensitive to noise). High K = high bias (inflexible) + low variance (stable). RULE OF THUMB: K = √(n_samples). For 150 iris → K ≈ 12. Always use ODD K (prevents tie votes in binary classification). WHEN KNN IS GOOD: Small datasets, few features, non-linear boundaries. WHEN KNN IS BAD: Large datasets (prediction is O(n) — slow). High dimensions (curse of dimensionality — all points are far apart). Need fast prediction (KNN stores ALL data, no compression).