🚀 Step 6 of 10
← Previous: Step 5 — File Upload + Pickle — Upload a Model
→
Benchmark Engine — Run a Model on a Dataset
Load the uploaded .pkl, load a holdout dataset, call model.predict(), compute accuracy/F1/RMSE. Save the result. This is the core of the platform.
🎯 Goal — what you'll achieve in this step
Upload a model, click 'Benchmark', and see an accuracy score. The score is saved to the DB and shown on the model's page.
📖 The basic concept
Learn this first. It's the pure Python/HTML idea, stripped of all app complexity.
# The basic concept: evaluate a model on held-out data
import pickle, numpy as np
from sklearn.metrics import accuracy_score
# Load the model
with open('model.pkl', 'rb') as f:
model = pickle.load(f)
# Load a holdout dataset (X_test, y_test)
X_test = np.array([[1], [2], [3], [4], [5]])
y_test = np.array([2, 4, 6, 8, 10]) # true labels
# Predict
predictions = model.predict(X_test)
print('predictions:', predictions) # [2. 4. 6. 8. 10.]
# Score (classification: accuracy; regression: MSE/R^2)
accuracy = accuracy_score(y_test, predictions.round())
print(f'accuracy = {accuracy:.3f}')
# The model never saw X_test during training — this is an honest score.
# Scoring on training data is cheating (overfitting).
🧮 Apply it to OpenBenchML
benchmark.py
python
# Step 6: The benchmark engine
import pickle
import numpy as np
from pathlib import Path
from sklearn.metrics import accuracy_score, mean_squared_error, r2_score
def run_benchmark(model_path: str, dataset: dict) -> dict:
'''
model_path: path to a .pkl file containing a trained model
dataset: {'X_test': np.array, 'y_test': np.array, 'task': 'classification'|'regression'}
Returns: {'score': float, 'metric': str, 'predictions': list}
'''
# 1. Load the model
with open(model_path, 'rb') as f:
model = pickle.load(f)
# 2. Predict on the holdout set
predictions = model.predict(dataset['X_test'])
# 3. Score based on task type
if dataset['task'] == 'classification':
score = accuracy_score(dataset['y_test'], predictions)
metric = 'accuracy'
else: # regression
score = r2_score(dataset['y_test'], predictions)
metric = 'r2'
return {
'score': float(score),
'metric': metric,
'n_samples': len(dataset['X_test']),
}
# A built-in dataset for testing
SAMPLE_DATASET = {
'X_test': np.array([[i] for i in range(100, 200)]),
'y_test': np.array([2 * i + 1 for i in range(100, 200)]),
'task': 'regression',
}
main.py
python
# Step 6: Add /benchmark/{model_id} on top of steps 1-5
from benchmark import run_benchmark, SAMPLE_DATASET
from database import Model, BenchmarkJob # add BenchmarkJob to models
# Add to database.py:
# class BenchmarkJob(Base):
# __tablename__ = 'benchmark_jobs'
# id = Column(Integer, primary_key=True)
# model_id = Column(Integer, ForeignKey('models.id'))
# score = Column(Float)
# metric = Column(String)
# created_at = Column(DateTime, default=datetime.utcnow)
@app.post('/benchmark/{model_id}')
async def benchmark_model(model_id: int, db: Session = Depends(get_db)):
# 1. Find the model
model = db.query(Model).filter(Model.id == model_id).first()
if not model:
return {'error': 'model not found'}
# 2. Run the benchmark
model_path = UPLOAD_DIR / model.filename
result = run_benchmark(str(model_path), SAMPLE_DATASET)
# 3. Save the result
job = BenchmarkJob(
model_id=model.id,
score=result['score'],
metric=result['metric'],
)
db.add(job); db.commit()
# 4. Return the score
return {'model': model.name, 'score': result['score'], 'metric': result['metric']}
How it works
Benchmarking is the scientific part of the platform. The flow: (1) Load the uploaded .pkl — pickle.load brings the trained model back into memory. (2) Load a holdout dataset that the model never saw during training. This is critical: scoring on training data is like grading a student on the exact questions they studied — it tells you nothing about real understanding. (3) model.predict(X_test) generates predictions. (4) Compare predictions to y_test using a metric: accuracy for classification (what fraction was correct), R² for regression (how much variance the model explains). (5) Save the result as a BenchmarkJob row. The key insight: the benchmark is reproducible — same model + same dataset = same score, every time. This is what makes leaderboards fair.
✏️ Try yourself
Add a second dataset (classification) with make_classification from sklearn. Detect the task type automatically by checking if y_test is integer-valued. Add precision and recall metrics for classification.
⚠️ Common mistakes to avoid
- Scoring on training data — the model memorized it, so the score is artificially high. Always hold out a test set the model never saw.
- Not handling predict() shape — some models return (n, 1), others (n,). Use `.ravel()` or `.flatten()` to be safe.
- Loading the model on every request — pickle.load is slow (~100ms). Cache the loaded model in a dict keyed by filename.
- Not catching exceptions in predict() — a malformed model will crash the route. Wrap in try/except and return a 400 with the error message.
🔍 Where this is used in OpenBenchML
app/routes/benchmark.py — /benchmark/{id} route. app/benchmark_engine/ — evaluator.py computes metrics, loader.py loads datasets.
Next — Step 7
Leaderboard — Rank Models by Score