Concepts 🚀 Project 🧮 Labs

Encoding — OneHot vs Label vs Target vs Frequency vs Embedding

Categorical encoding is NOT just pd.get_dummies(). 5 strategies, each for a different situation. Pick wrong = garbage model.

script.py v3.1

import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder, LabelEncoder, OrdinalEncoder
from sklearn.model_selection import train_test_split

# ── Dataset with 3 types of categorical columns ──
np.random.seed(42)
n = 1000
df = pd.DataFrame({
    # LOW cardinality (3 values) — OneHot is perfect
    'color': np.random.choice(['red', 'green', 'blue'], n),
    # ORDINAL (has natural order) — Label/Ordinal encoding is correct
    'education': np.random.choice(['High School', 'Bachelor', 'Master', 'PhD'], n),
    # HIGH cardinality (50 values) — OneHot creates 50 columns (curse of dimensionality)
    'city': np.random.choice([f'city_{i}' for i in range(50)], n),
    # TARGET (what we predict)
    'income': np.random.lognormal(10.5, 0.5, n),
})

print('Dataset:')
print(df.head())
print(f'\nCardinalities: color={df["color"].nunique()}, education={df["education"].nunique()}, city={df["city"].nunique()}')

# ════════════════════════════════════════════════════════════════════
#  STRATEGY 1: OneHotEncoder (for LOW cardinality, NOMINAL categories)
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('STRATEGY 1: OneHotEncoder (color — 3 values, no order)')
print('='*60)
ohe = OneHotEncoder(drop='first', sparse_output=False)  # drop='first' avoids multicollinearity
color_encoded = ohe.fit_transform(df[['color']])
print(f'  Before: 1 column (color)')
print(f'  After: {color_encoded.shape[1]} columns: {list(ohe.get_feature_names_out())}')
print(f'  WHY: color has no natural order. red=1, blue=2 would imply red < blue.')
print(f'  OneHot creates binary columns — no implied order.')
print(f'  drop="first" drops one column (redundant) — avoids the dummy variable trap.')

# ════════════════════════════════════════════════════════════════════
#  STRATEGY 2: OrdinalEncoder (for ORDINAL categories — has natural order)
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('STRATEGY 2: OrdinalEncoder (education — High School < Bachelor < Master < PhD)')
print('='*60)
# MUST specify the order explicitly!
education_order = [['High School', 'Bachelor', 'Master', 'PhD']]
oe = OrdinalEncoder(categories=education_order)
edu_encoded = oe.fit_transform(df[['education']])
print(f'  Mapping: High School=0, Bachelor=1, Master=2, PhD=3')
print(f'  First 5: {edu_encoded[:5].ravel()}')
print(f'  WHY: education HAS order. PhD > Bachelor makes sense.')
print(f'  OneHot would lose this order information.')
print(f'  WARNING: only use when the order is REAL. Don\'t use for nominal (color).')

# ════════════════════════════════════════════════════════════════════
#  STRATEGY 3: Frequency encoding (for HIGH cardinality)
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('STRATEGY 3: Frequency encoding (city — 50 values, too many for OneHot)')
print('='*60)
city_freq = df['city'].value_counts(normalize=True)
df['city_freq'] = df['city'].map(city_freq)
print(f'  city_0 appears {city_freq["city_0"]*100:.1f}% of the time → encoded as {city_freq["city_0"]:.4f}')
print(f'  city_1 appears {city_freq["city_1"]*100:.1f}% of the time → encoded as {city_freq["city_1"]:.4f}')
print(f'  WHY: OneHot would create 50 columns (curse of dimensionality).')
print(f'  Frequency encoding = 1 column. Tree models (RF, XGBoost) handle this well.')
print(f'  LOSES information: 2 cities with same frequency get the same code.')

# ════════════════════════════════════════════════════════════════════
#  STRATEGY 4: Target encoding (for HIGH cardinality + when target matters)
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('STRATEGY 4: Target encoding (city → average income per city)')
print('='*60)
# WARNING: must compute on TRAIN only (else data leakage!)
X_train, X_test, y_train, y_test = train_test_split(df[['city']], df['income'], test_size=0.2, random_state=42)
target_mean = pd.concat([X_train, y_train], axis=1).groupby('city')['income'].mean()
# Smooth to avoid overfitting on rare cities
overall_mean = y_train.mean()
city_counts = X_train['city'].value_counts()
smoothed = (target_mean * city_counts + overall_mean * 10) / (city_counts + 10)
print(f'  city_0 mean income: {target_mean["city_0"]:.0f}')
print(f'  city_1 mean income: {target_mean["city_1"]:.0f}')
print(f'  WHY: encodes city by its PREDICTIVE power.')
print(f'  WARNING: HIGH risk of overfitting. Use smoothing + cross-validation.')
print(f'  SMOOTHING: (mean * count + global_mean * 10) / (count + 10)')
print(f'  Pulls rare cities toward the global mean — prevents overfitting.')

# ════════════════════════════════════════════════════════════════════
#  STRATEGY 5: Just drop the column (sometimes the best choice)
# ════════════════════════════════════════════════════════════════════
print('\n' + '='*60)
print('STRATEGY 5: Drop (when cardinality > 95% or column is useless)')
print('='*60)
print(f'  If city had 950 unique values out of 1000 rows...')
print(f'  ...each city appears ~1 time. Encoding is meaningless.')
print(f'  Drop the column. ML can\'t learn from near-unique identifiers.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

5 encoding strategies — pick by cardinality + order: ONEHOT — low cardinality (<10), nominal (no order). drop='first' to avoid multicollinearity. ORDINAL — when categories HAVE a natural order (Low < Med < High). Specify the order explicitly! FREQUENCY — high cardinality (10-100). Replace category with its frequency. Good for trees. TARGET — high cardinality + predictive target. Replace category with mean target. HIGH overfit risk — use smoothing + CV. DROP — cardinality > 95% (e.g., user_id, email). Useless for ML. RULE OF THUMB: < 10 unique values → OneHot. 10-100 → Frequency or Target. > 100 with predictive power → Target (with smoothing). > 95% unique → Drop. Has natural order → Ordinal. Common mistake: using LabelEncoder on nominal data. LabelEncoder implies order (red=0 < blue=1 < green=2) which is FALSE.

✏️ Try these changes

Remove drop='first' from OneHotEncoder
→ creates 3 columns instead of 2 — multicollinearity risk for linear models
Use OrdinalEncoder WITHOUT specifying categories (default alphabetical)
→ Bachelor=0, High School=1, Master=2, PhD=3 — WRONG order!
Increase city cardinality to 200: np.random.choice([f'city_{i}' for i in range(200)], n)
→ OneHot would create 200 columns — frequency/target encoding better
Change smoothing factor from 10 to 100
→ more smoothing — rare cities pulled harder toward global mean

← Previous lab

Imputation — when to use mean vs median vs mode vs KNN vs MICE

Next lab →

EDA mindset — 8 questions that reveal what the data is hiding