Handle ANY model type (sklearn, xgboost, custom)

A benchmarking engine must work with ANY .pkl — not just LinearRegression. Use duck typing: check for .predict().

script.py v3.1

import joblib
import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.svm import SVR
from sklearn.datasets import make_regression, make_classification
from sklearn.model_selection import train_test_split

# ── Train 4 different models + save them ──
# Regression models
Xr, yr = make_regression(n_samples=200, n_features=5, noise=10, random_state=42)
joblib.dump(LinearRegression().fit(Xr, yr), 'lr.pkl')
joblib.dump(RandomForestRegressor(n_estimators=20, random_state=42).fit(Xr, yr), 'rf_reg.pkl')
joblib.dump(SVR().fit(Xr, yr), 'svr.pkl')

# Classification model
Xc, yc = make_classification(n_samples=200, n_features=5, n_informative=3, random_state=42)
joblib.dump(LogisticRegression().fit(Xc, yc), 'logreg.pkl')

# ── BENCHMARK ENGINE: load any .pkl + figure out what it is ──
def benchmark_any_model(pkl_path, X_test, y_test=None):
    """Load a .pkl model and benchmark it — works with ANY model type."""
    model = joblib.load(pkl_path)

    # Duck typing: check what methods the model has
    has_predict = hasattr(model, 'predict')
    has_predict_proba = hasattr(model, 'predict_proba')
    has_feature_importances = hasattr(model, 'feature_importances_')
    has_coef = hasattr(model, 'coef_')

    print(f'\nLoaded {pkl_path}:')
    print(f'  Type: {type(model).__name__}')
    print(f'  Module: {type(model).__module__}')
    print(f'  has predict:            {has_predict}')
    print(f'  has predict_proba:      {has_predict_proba}')
    print(f'  has feature_importances: {has_feature_importances}')
    print(f'  has coef_:              {has_coef}')

    if not has_predict:
        print('  ERROR: model has no predict() method — not a valid ML model')
        return None

    # Run prediction
    y_pred = model.predict(X_test)
    print(f'  Prediction shape: {y_pred.shape}')
    print(f'  Prediction dtype: {y_pred.dtype}')
    print(f'  First 3 predictions: {y_pred[:3]}')

    # Detect regression vs classification
    # If predictions are floats → regression. If integers → classification.
    is_regression = y_pred.dtype.kind == 'f'
    print(f'  Task type: {"REGRESSION" if is_regression else "CLASSIFICATION"}')

    return model

# Test with all 4 models
print('Benchmarking 4 different model types:')
print('=' * 60)
benchmark_any_model('lr.pkl', Xr[:10])
benchmark_any_model('rf_reg.pkl', Xr[:10])
benchmark_any_model('svr.pkl', Xr[:10])
benchmark_any_model('logreg.pkl', Xc[:10])

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

A benchmarking engine MUST work with any model — not just the ones you trained. Use DUCK TYPING: check hasattr(model, 'predict') instead of isinstance(model, LinearRegression). Why? Because xgboost.XGBRegressor, lightgbm.LGBMRegressor, custom classes — all have .predict(). Don't care WHAT it is, care what it CAN DO. Detect regression vs classification by checking y_pred.dtype: float → regression (predict returns continuous values). int → classification (predict returns class labels). Also check for predict_proba — only classifiers have it. feature_importances_ → tree-based models (RF, XGBoost). coef_ → linear models (LinearRegression, LogisticRegression, SVM). This is how OpenBenchML handles ANY uploaded .pkl file.

✏️ Try these changes

Add a dummy class with no predict() method and try to benchmark it
→ shows the error handling
Change the regression detection to also check predict_proba
→ classifiers have predict_proba, regressors don't
Add from sklearn.cluster import KMeans; joblib.dump(KMeans(n_clusters=3).fit(Xr), 'kmeans.pkl'); benchmark_any_model('kmeans.pkl', Xr[:10])
→ KMeans has predict() but it's clustering, not classification
Print model.get_params() if the model has it
→ shows hyperparameters

← Previous lab

Cross-validation — more honest than a single split

Next lab →

FastAPI endpoint — accept .pkl file upload