Save benchmark results + compare models

Run benchmarks on multiple models, save results to a DataFrame, compare side-by-side. This is how leaderboards work.

script.py v3.1

import pickle
import time
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.svm import SVR
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

# Generate data
X, y = make_regression(n_samples=1000, n_features=10, noise=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# ── Define models to benchmark ──
models = {
    'LinearRegression': LinearRegression(),
    'Ridge': Ridge(alpha=1.0),
    'RandomForest': RandomForestRegressor(n_estimators=50, random_state=42),
    'GradientBoosting': GradientBoostingRegressor(n_estimators=50, random_state=42),
    'SVR': SVR(),
}

# ── Benchmark function (simplified) ──
def benchmark(model, X_train, y_train, X_test, y_test):
    # Train
    t0 = time.perf_counter()
    model.fit(X_train, y_train)
    train_time = time.perf_counter() - t0

    # Predict
    t0 = time.perf_counter()
    y_pred = model.predict(X_test)
    infer_time = time.perf_counter() - t0

    # Metrics
    mae = mean_absolute_error(y_test, y_pred)
    r2 = r2_score(y_test, y_pred)

    return {
        'train_time_ms': train_time * 1000,
        'infer_time_ms': infer_time * 1000,
        'throughput_sps': len(X_test) / infer_time,
        'MAE': mae,
        'R2': r2,
    }

# ── Run benchmark on all models ──
results = []
for name, model in models.items():
    print(f'Benchmarking {name}...')
    metrics = benchmark(model, X_train, y_train, X_test, y_test)
    metrics['model'] = name
    results.append(metrics)

# ── Save to DataFrame (this is how leaderboards work) ──
df = pd.DataFrame(results).set_index('model')
print('\n' + '=' * 70)
print('BENCHMARK RESULTS')
print('=' * 70)
print(df.round(3).to_string())

# ── Find the winner ──
best_r2 = df['R2'].idxmax()
fastest = df['infer_time_ms'].idxmin()
print(f'\nBest R²:  {best_r2} ({df.loc[best_r2, "R2"]:.4f})')
print(f'Fastest:  {fastest} ({df.loc[fastest, "infer_time_ms"]:.2f} ms)')

# ── Save results to CSV (persist for leaderboard) ──
df.to_csv('benchmark_results.csv')
print('\nSaved -> benchmark_results.csv')

# ── Load + display later (simulating a leaderboard) ──
print('\nLeaderboard (sorted by R²):')
leaderboard = pd.read_csv('benchmark_results.csv').sort_values('R2', ascending=False)
print(leaderboard[['model', 'R2', 'MAE', 'infer_time_ms']].round(3).to_string(index=False))

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

This is how a LEADERBOARD works: (1) Define multiple models to compare. (2) Benchmark each — train time, inference time, metrics. (3) Store results in a pandas DataFrame. (4) Save to CSV (or a database table). (5) Load + sort by metric to show the leaderboard. The benchmark function is the SAME for all models — that's the point. You can add XGBoost, LightGBM, neural nets — same function, same DataFrame, same leaderboard. In OpenBenchML, the DataFrame is stored in a database (LeaderboardEntry table) and rendered as an HTML table on the /leaderboard page. WebSocket pushes update it live.

✏️ Try these changes

Add from sklearn.neighbors import KNeighborsRegressor; models['KNN'] = KNeighborsRegressor(n_neighbors=5)
→ another model in the comparison
Change the sort from R2 to MAE (ascending=True for errors)
→ leaderboard sorted by lowest error
Add a 'memory_mb' column using psutil.Process().memory_info().rss
→ tracks RAM usage per model
Wrap the benchmark loop in cross_validate for more honest results
→ 5-fold CV per model

← Previous lab

Full benchmark engine — putting it all together

Next lab →

Error handling — when the .pkl is broken or wrong