Concepts 🚀 Project 🧮 Labs

Full ML workflow — the senior data scientist's process

Not 'load + fit + predict'. The REAL workflow: 10 steps from raw data to production-ready model. This is what senior DS actually do.

script.py v3.1

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, cross_validate, KFold
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_absolute_error, r2_score
import time

# ════════════════════════════════════════════════════════════════════
#  STEP 1: LOAD + UNDERSTAND THE DATA
# ════════════════════════════════════════════════════════════════════
print('STEP 1: LOAD + UNDERSTAND')
print('-' * 50)
# Synthetic dataset (in real life: pd.read_csv('data.csv'))
np.random.seed(42)
n = 1000
df = pd.DataFrame({
    'age': np.random.normal(40, 12, n).clip(18, 80),
    'income': np.random.lognormal(10.5, 0.6, n),
    'years_experience': np.random.randint(0, 40, n),
    'education': np.random.choice(['High School', 'Bachelor', 'Master', 'PhD'], n),
    'city': np.random.choice(['NYC', 'LA', 'SF', 'Chicago', 'Boston'], n),
})
df['salary'] = (
    30000 + df['years_experience'] * 2500 +
    df['education'].map({'High School': 0, 'Bachelor': 4000, 'Master': 8000, 'PhD': 12000}) +
    np.random.normal(0, 8000, n) +
    df['city'].map({'NYC': 15000, 'SF': 20000, 'LA': 10000, 'Chicago': 5000, 'Boston': 8000})
)
# Inject real-world messiness
df.loc[df.sample(50, random_state=42).index, 'age'] = np.nan
print(f'Shape: {df.shape}')
print(f'Nulls: {df.isnull().sum().sum()}')
print(f'Target: salary (continuous → regression)')

# ════════════════════════════════════════════════════════════════════
#  STEP 2: SEPARATE X (features) AND y (target) — BEFORE any processing
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 2: X/y SPLIT')
print('-' * 50)
X = df.drop(columns=['salary'])
y = df['salary']
print(f'X shape: {X.shape} (features)')
print(f'y shape: {y.shape} (target)')

# ════════════════════════════════════════════════════════════════════
#  STEP 3: TRAIN/TEST SPLIT — BEFORE preprocessing (prevents leakage)
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 3: TRAIN/TEST SPLIT (BEFORE preprocessing)')
print('-' * 50)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f'Train: {len(X_train)} rows')
print(f'Test:  {len(X_test)} rows')
print('WHY before preprocessing? If you fit the scaler on ALL data,')
print('test data leaks into the scaler → optimistic scores.')

# ════════════════════════════════════════════════════════════════════
#  STEP 4: IDENTIFY COLUMN TYPES
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 4: COLUMN TYPES')
print('-' * 50)
numerical_features = ['age', 'income', 'years_experience']
categorical_nominal = ['city']  # no natural order
categorical_ordinal = ['education']  # has order: HS < Bachelor < Master < PhD
print(f'Numerical:   {numerical_features}')
print(f'Nominal:     {categorical_nominal} (OneHot)')
print(f'Ordinal:     {categorical_ordinal} (OrdinalEncoder with explicit order)')

# ════════════════════════════════════════════════════════════════════
#  STEP 5: BUILD PREPROCESSING PIPELINES
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 5: PREPROCESSING PIPELINES')
print('-' * 50)
numeric_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])
nominal_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(drop='first', handle_unknown='ignore'))
])
ordinal_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OrdinalEncoder(categories=[['High School', 'Bachelor', 'Master', 'PhD']]))
])
preprocessor = ColumnTransformer([
    ('num', numeric_pipeline, numerical_features),
    ('nom', nominal_pipeline, categorical_nominal),
    ('ord', ordinal_pipeline, categorical_ordinal),
])
print('Built 3 pipelines: numeric (impute+scale), nominal (impute+OneHot), ordinal (impute+OrdinalEncoder)')

# ════════════════════════════════════════════════════════════════════
#  STEP 6: CREATE FULL PIPELINE (preprocessing + model)
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 6: FULL PIPELINE')
print('-' * 50)
model = Pipeline([
    ('preprocessing', preprocessor),
    ('regressor', RandomForestRegressor(n_estimators=50, random_state=42))
])
print('Pipeline: preprocessing → RandomForestRegressor')

# ════════════════════════════════════════════════════════════════════
#  STEP 7: TRAIN + EVALUATE
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 7: TRAIN + EVALUATE')
print('-' * 50)
t0 = time.perf_counter()
model.fit(X_train, y_train)
train_time = time.perf_counter() - t0

y_train_pred = model.predict(X_train)
y_test_pred = model.predict(X_test)

print(f'Training time: {train_time:.2f}s')
print(f'Train R²: {r2_score(y_train, y_train_pred):.4f}')
print(f'Test  R²: {r2_score(y_test, y_test_pred):.4f}')
print(f'Train MAE: {mean_absolute_error(y_train, y_train_pred):.0f}')
print(f'Test  MAE: {mean_absolute_error(y_test, y_test_pred):.0f}')
print(f'Gap (overfitting?): {r2_score(y_train, y_train_pred) - r2_score(y_test, y_test_pred):.4f}')

# ════════════════════════════════════════════════════════════════════
#  STEP 8: CROSS-VALIDATION (more honest than single split)
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 8: CROSS-VALIDATION')
print('-' * 50)
cv = KFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(model, X, y, cv=cv, scoring=['r2', 'neg_mean_absolute_error'])
print(f'CV R²:   {cv_results["test_r2"].mean():.4f} ± {cv_results["test_r2"].std():.4f}')
print(f'CV MAE:  {-cv_results["test_neg_mean_absolute_error"].mean():.0f}')

# ════════════════════════════════════════════════════════════════════
#  STEP 9: COMPARE MULTIPLE MODELS
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 9: COMPARE MODELS')
print('-' * 50)
models_to_try = {
    'LinearRegression': Pipeline([('prep', preprocessor), ('model', LinearRegression())]),
    'Ridge': Pipeline([('prep', preprocessor), ('model', Ridge(alpha=1.0))]),
    'RandomForest': model,  # already built
}
for name, m in models_to_try.items():
    scores = cross_validate(m, X, y, cv=cv, scoring='r2')['test_r2']
    print(f'  {name:20s}  R² = {scores.mean():.4f} ± {scores.std():.4f}')

# ════════════════════════════════════════════════════════════════════
#  STEP 10: SAVE THE WINNER + DEPLOY
# ════════════════════════════════════════════════════════════════════
print('\nSTEP 10: SAVE + DEPLOY')
print('-' * 50)
import joblib
joblib.dump(model, 'best_model.pkl')
print('Saved -> best_model.pkl')
print('The pipeline INCLUDES preprocessing — no need to remember the scaler.')
print('In production: model = joblib.load("best_model.pkl"); model.predict(new_data)')
print('\nFULL WORKFLOW COMPLETE. This is what senior data scientists do.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

10 steps = the real ML workflow. NOT 'load + fit + predict'. STEP 1: UNDERSTAND — shape, nulls, types, distributions, leaks. STEP 2: X/y SPLIT — separate features from target BEFORE processing. STEP 3: TRAIN/TEST SPLIT — BEFORE preprocessing to prevent leakage. If you scale on all data, test data leaks into the scaler → fake high scores. STEP 4: COLUMN TYPES — numerical, nominal (no order), ordinal (has order). STEP 5: PIPELINES — one per type. Impute + scale for numeric, impute + encode for categorical. STEP 6: FULL PIPELINE — chain preprocessing + model. One fit(), one predict(). STEP 7: TRAIN + EVALUATE — report TRAIN + TEST scores. Gap = overfitting. STEP 8: CROSS-VALIDATION — 5-fold CV. More honest than single split. STEP 9: COMPARE MODELS — never trust one model. Try 3+, pick the winner. STEP 10: SAVE THE PIPELINE — preprocessing is BAKED IN. No need to remember the scaler. WHY THIS MATTERS: skip any step = model fails in production. Skip step 3 = data leakage = 99% train, 50% production. Skip step 5 = manual preprocessing = forget a step = crash. Skip step 8 = lucky split = model worse than you think. Skip step 9 = stuck with a bad model = miss easy accuracy. This is the difference between 'I trained a model' (fresher) and 'I built a production ML system' (senior).

✏️ Try these changes

Change the model from RandomForestRegressor to GradientBoostingRegressor
→ often higher accuracy on tabular data
Add a leak column: df['salary_per_year'] = df['salary'] / df['years_experience'].clip(1)
→ Step 1 should detect it (high correlation with target)
Change the imputation strategy from 'median' to 'mean'
→ income is skewed — median is better, mean is pulled by outliers
Add Step 9b: hyperparameter tuning with GridSearchCV
→ find the best n_estimators for RandomForest

← Previous lab

Overfitting — the #1 reason ML models fail in production

Next lab →

Logistic Regression visualized — sigmoid + decision boundary (iris)