Probability for ML — Bayes, distributions, maximum likelihood

ML is applied probability. Understand Bayes' theorem, common distributions, and maximum likelihood estimation.

script.py v3.1

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt

# ════════════════════════════════════════════════════════════════════
#  1. BAYES' THEOREM — the foundation of probabilistic ML
# ════════════════════════════════════════════════════════════════════
print('='*60)
print('1. BAYES\' THEOREM')
print('='*60)
print('P(A|B) = P(B|A) * P(A) / P(B)')
print()
print('Example: spam detection')
print('  A = email is spam')
print('  B = email contains "FREE"')
print('  P(A) = 0.3            (30% of emails are spam)')
print('  P(B|A) = 0.8          (80% of spam contains "FREE")')
print('  P(B|¬A) = 0.1         (10% of non-spam contains "FREE")')
print('  P(B) = P(B|A)*P(A) + P(B|¬A)*P(¬A) = 0.8*0.3 + 0.1*0.7 = 0.31')
print()

p_a = 0.3
p_b_given_a = 0.8
p_b_given_not_a = 0.1
p_b = p_b_given_a * p_a + p_b_given_not_a * (1 - p_a)
p_a_given_b = p_b_given_a * p_a / p_b
print(f'  P(spam | "FREE") = {p_a_given_b:.3f}')
print(f'  Even though 80% of spam has "FREE", only {p_a_given_b*100:.1f}% of "FREE" emails are spam.')
print(f'  This is the BASE RATE FALLACY — people ignore P(A).')
print(f'  Naive Bayes classifier uses this for EVERY feature.')

# ════════════════════════════════════════════════════════════════════
#  2. COMMON DISTRIBUTIONS — what shape is your data?
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('2. COMMON DISTRIBUTIONS')
print('='*60)
fig, axes = plt.subplots(2, 3, figsize=(15, 8))

# Normal (Gaussian) — height, age, errors
x = np.linspace(-5, 5, 100)
axes[0,0].plot(x, stats.norm.pdf(x, 0, 1), color='#a0c000', linewidth=2)
axes[0,0].set_title('Normal (Gaussian)\nheight, age, errors')
axes[0,0].axvline(0, color='gray', linestyle='--', alpha=0.5)

# Lognormal — income, prices, word frequencies
x = np.linspace(0, 5, 100)
axes[0,1].plot(x, stats.lognorm.pdf(x, 0.5), color='#58a6ff', linewidth=2)
axes[0,1].set_title('Lognormal\nincome, prices, word freq')

# Uniform — random numbers, dice
x = np.linspace(0, 1, 100)
axes[0,2].plot(x, stats.uniform.pdf(x), color='#bc8cff', linewidth=2)
axes[0,2].set_title('Uniform\ndice, random sampling')

# Bernoulli — coin flip, binary outcome
axes[1,0].bar([0, 1], [0.3, 0.7], color='#f85149')
axes[1,0].set_title('Bernoulli\ncoin flip, click/no-click')
axes[1,0].set_xticks([0, 1])

# Binomial — n coin flips
x = np.arange(0, 11)
axes[1,1].bar(x, stats.binom.pmf(x, 10, 0.5), color='#ffa657')
axes[1,1].set_title('Binomial\n# heads in 10 flips')

# Poisson — events per time period
x = np.arange(0, 15)
axes[1,2].bar(x, stats.poisson.pmf(x, 3), color='#3fb950')
axes[1,2].set_title('Poisson\n# emails per hour')

plt.tight_layout()
plt.savefig('distributions.png', dpi=100, bbox_inches='tight')
plt.show()
print('Normal: symmetric, bell-shaped. Height, age, measurement errors.')
print('Lognormal: right-skewed. Income, prices, city populations.')
print('Uniform: flat. Random sampling, dice rolls.')
print('Bernoulli: 0 or 1. Click/no-click, spam/not-spam.')
print('Binomial: count of successes. # heads in N flips.')
print('Poisson: count of events. # emails per hour, # accidents per month.')

# ════════════════════════════════════════════════════════════════════
#  3. MAXIMUM LIKELIHOOD ESTIMATION (MLE)
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('3. MAXIMUM LIKELIHOOD ESTIMATION (MLE)')
print('='*60)
print('Idea: find the parameters that make the observed data MOST LIKELY.')
print()
print('Example: coin flipped 10 times, 7 heads. Is it fair?')
print('  Likelihood: P(data | p) = C(10,7) * p^7 * (1-p)^3')
print('  Maximize over p: take derivative, set to 0.')
print('  Result: p = 7/10 = 0.7 (the sample proportion)')

# Plot the likelihood function
p_values = np.linspace(0, 1, 100)
likelihood = p_values**7 * (1 - p_values)**3
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(p_values, likelihood, color='#a0c000', linewidth=2)
ax.axvline(0.7, color='red', linestyle='--', label=f'MLE: p=0.7')
ax.set_xlabel('p (probability of heads)')
ax.set_ylabel('Likelihood')
ax.set_title('Likelihood function — MLE finds the peak')
ax.legend()
plt.tight_layout()
plt.savefig('mle.png', dpi=100, bbox_inches='tight')
plt.show()
print(f'\nMLE: p = {0.7} — the peak of the likelihood function.')
print()
print('LinearRegression = MLE of Gaussian noise.')
print('LogisticRegression = MLE of Bernoulli outcome.')
print('Neural Network (cross-entropy loss) = MLE of categorical outcome.')
print('MLE is the bridge between probability theory and ML training.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

ML = applied probability. 3 concepts to know: BAYES' THEOREM: P(A|B) = P(B|A)*P(A)/P(B). Update beliefs with evidence. Used in: Naive Bayes, Bayesian inference, A/B testing. Base rate fallacy: people ignore P(A). 80% of spam has 'FREE' ≠ most 'FREE' emails are spam. DISTRIBUTIONS: know the shape of your data. Normal — symmetric (height, age, errors). Lognormal — right-skewed (income, prices). Log-transform to make normal. Uniform — flat (random sampling). Bernoulli — 0/1 (click/no-click). Binomial — count of successes (# heads in N flips). Poisson — count of events per period (# emails/hour). MLE (Maximum Likelihood Estimation): find params that make data MOST LIKELY. LinearRegression = MLE of Gaussian noise. LogisticRegression = MLE of Bernoulli. Cross-entropy loss = MLE of categorical. MLE is the bridge: probability theory → ML training. Understand these 3 = understand why ML models use the loss functions they do.

✏️ Try these changes

Change P(A) from 0.3 to 0.5
→ P(spam|FREE) jumps — base rate matters
Change the Bernoulli p from 0.7 to 0.3
→ bar chart flips — more 0s than 1s
Change Poisson lambda from 3 to 10
→ distribution shifts right — more events per period
Change the MLE example: 70 heads in 100 flips
→ MLE = 0.7 still, but confidence higher

← Previous lab

Gradient Descent — the engine behind neural networks

Next lab →

Model selection — which algorithm for which problem?