Concepts 🚀 Project 🧮 Labs

PCA visualized — variance + projection (iris + digits)

SEE PCA reduce 4D iris to 2D. SEE the scree plot. SEE 64D digits collapse to 2D. The geometry of dimensionality reduction.

script.py v3.1

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris, load_digits
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# ════════════════════════════════════════════════════════════════════
#  PART 1: PCA on iris (4D → 2D) — SEE the projection
# ════════════════════════════════════════════════════════════════════
iris = load_iris()
X = iris.data  # 4 features: sepal length, sepal width, petal length, petal width
y = iris.target

# ALWAYS scale before PCA (PCA is sensitive to feature scale)
X_scaled = StandardScaler().fit_transform(X)

# Fit PCA — reduce 4D to 2D
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print('PCA on iris (4D → 2D):')
print(f'  Original shape: {X_scaled.shape}')
print(f'  After PCA:      {X_pca.shape}')
print(f'  Variance explained: {pca.explained_variance_ratio_.round(3)}')
print(f'  Total variance captured: {pca.explained_variance_ratio_.sum():.1%}')

fig, axes = plt.subplots(1, 3, figsize=(18, 5))

# PLOT 1: Scree plot — how much variance each component captures
components = range(1, len(pca.explained_variance_ratio_) + 1)
axes[0].bar(components, pca.explained_variance_ratio_, color='#a0c000', alpha=0.7)
axes[0].set_xlabel('Principal Component')
axes[0].set_ylabel('Variance Explained')
axes[0].set_title('Scree Plot — which components matter?')
for i, v in enumerate(pca.explained_variance_ratio_):
    axes[0].text(i + 1, v + 0.02, f'{v:.1%}', ha='center', fontsize=10)

# PLOT 2: 2D projection — the whole point of PCA
colors = ['#e74c3c', '#2ecc71', '#3498db']
for i, species in enumerate(iris.target_names):
    mask = y == i
    axes[1].scatter(X_pca[mask, 0], X_pca[mask, 1], c=colors[i],
                   label=species, edgecolors='black', s=40)
axes[1].set_xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%} variance)')
axes[1].set_ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%} variance)')
axes[1].set_title('Iris projected to 2D (from 4D)')
axes[1].legend()

# PLOT 3: PCA on digits (64D → 2D) — the power of dimensionality reduction
digits = load_digits()
X_digits = digits.data  # 64 features (8x8 pixel images)
y_digits = digits.target

pca_digits = PCA(n_components=2)
X_digits_pca = pca_digits.fit_transform(X_digits)

scatter = axes[2].scatter(X_digits_pca[:, 0], X_digits_pca[:, 1],
                          c=y_digits, cmap='tab10', s=10, alpha=0.6)
axes[2].set_xlabel(f'PC1 ({pca_digits.explained_variance_ratio_[0]:.1%})')
axes[2].set_ylabel(f'PC2 ({pca_digits.explained_variance_ratio_[1]:.1%})')
axes[2].set_title(f'Digits 64D → 2D\n({pca_digits.explained_variance_ratio_[:2].sum():.1%} variance captured)')
plt.colorbar(scatter, ax=axes[2], label='Digit (0-9)')

plt.tight_layout()

print(f'\nPCA on digits (64D → 2D):')
print(f'  Original: 64 features (8×8 pixels)')
print(f'  After PCA: 2 features')
print(f'  Variance captured: {pca_digits.explained_variance_ratio_[:2].sum():.1%}')
print(f'  (Only ~28% of variance — but clusters are still visible!)')
print()
print('THE MATHS:')
print('  PCA finds the directions (principal components) of MAXIMUM variance.')
print('  Step 1: Center the data (subtract mean)')
print('  Step 2: Compute covariance matrix Σ = (1/n) * XᵀX')
print('  Step 3: Eigendecomposition: Σ = VΛVᵀ')
print('  Step 4: Top k eigenvectors = the k principal components')
print('  Step 5: Project: X_projected = X · V[:,:k]')
print()
print('  Eigenvectors = directions of maximum variance')
print('  Eigenvalues = how much variance each direction captures')
print('  Scree plot shows eigenvalues — pick k where the curve "elbows"')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

PCA (Principal Component Analysis) — VISUALIZED: WHAT: reduces dimensions while keeping maximum variance. 4D → 2D for visualization. MATHS: (1) Center data (subtract mean). (2) Covariance matrix Σ = (1/n) XᵀX — shows how features vary together. (3) Eigendecomposition: Σ = VΛVᵀ. V = eigenvectors (directions), Λ = eigenvalues (variance). (4) Top k eigenvectors = the k principal components (directions of max variance). (5) Project: X_proj = X · V[:,:k]. New features = linear combinations of old. SCREE PLOT: bar chart of variance per component. Look for the 'elbow' — components after the elbow add little value. IRIS: 4D → 2D captures 95.8% of variance. Setosa is perfectly separated. Versicolor + virginica overlap slightly — they're genuinely similar. DIGITS: 64D → 2D captures only 28% of variance, but digit clusters are STILL VISIBLE. This is the power of PCA — even with massive dimensionality reduction, structure survives. ALWAYS scale before PCA! Without scaling, features with large values (like salary in thousands) dominate the variance, and PCA just finds them.

✏️ Try these changes

Change n_components from 2 to 3 for iris PCA
→ captures more variance — try plotting in 3D with ax.scatter3D
Remove StandardScaler (use raw X instead of X_scaled)
→ petal length (0-7cm) dominates — sepal width (2-4cm) is ignored
Add a 3rd subplot: PCA with 3 components on iris, colored by species
→ 3D scatter shows even better separation
Change load_digits to load_breast_cancer (30 features)
→ 30D → 2D — see how cancer/benign separate

← Previous lab

Logistic Regression visualized — sigmoid + decision boundary (iris)

Next lab →

SVM visualized — hyperplane + margins + support vectors (iris)