Measure inference time + throughput

How fast can the model make predictions? Measure inference time, per-sample latency, and throughput (samples/sec).

script.py v3.1

import time
import joblib
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression

# Generate test data
X, _ = make_regression(n_samples=10000, n_features=10, noise=10, random_state=42)
X_test = X[:1000]  # 1000 samples for inference benchmark
print(f'Test set: {X_test.shape[0]} samples, {X_test.shape[1]} features')

# Train + save models
models = {
    'LinearRegression': LinearRegression().fit(X, X[:, 0]),  # dummy target
    'RandomForest(50)': RandomForestRegressor(n_estimators=50, random_state=42).fit(X, X[:, 0]),
}

# ── Inference benchmark function ──
def benchmark_inference(model, X_test, name):
    # Warm-up: first predict() call is slower (lazy init)
    model.predict(X_test[:10])

    # Time the prediction
    start = time.perf_counter()
    y_pred = model.predict(X_test)
    end = time.perf_counter()

    inference_time = end - start
    n_samples = len(X_test)
    latency_per_sample = inference_time / n_samples
    throughput = n_samples / inference_time

    print(f'  {name:25s}  '
          f'total: {inference_time*1000:7.2f} ms  '
          f'latency: {latency_per_sample*1e6:7.1f} us/sample  '
          f'throughput: {throughput:8.0f} samples/sec')
    return inference_time, throughput

print('\nInference benchmark:')
print('-' * 80)
for name, model in models.items():
    benchmark_inference(model, X_test, name)

# ── Single-sample latency (what real-time APIs care about) ──
print('\nSingle-sample latency (real-time API scenario):')
for name, model in models.items():
    sample = X_test[:1]  # 1 sample, shape (1, 10)
    # Average over 100 calls
    start = time.perf_counter()
    for _ in range(100):
        model.predict(sample)
    end = time.perf_counter()
    avg_latency_us = (end - start) / 100 * 1e6
    print(f'  {name:25s}  {avg_latency_us:7.1f} us per prediction')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Three metrics matter for inference: (1) TOTAL inference time — how long for the whole test set. (2) PER-SAMPLE LATENCY — inference_time / n_samples. What users feel. (3) THROUGHPUT — n_samples / inference_time. How many requests/sec the model can handle. WARM-UP is critical: the first predict() call is slower (lazy initialization, JIT compilation). Always warm up before timing, or your numbers will be skewed. For real-time APIs, measure SINGLE-SAMPLE latency (predict 1 sample, average over 100 calls). Batch prediction is always faster per-sample (amortized overhead).

✏️ Try these changes

Change X_test size from 1000 to 10000
→ bigger batch → higher throughput
Change n_estimators from 50 to 200
→ RandomForest takes longer per prediction
Remove the warm-up call
→ first prediction will be slower, skewing the average
Change the single-sample loop from 100 to 1000 iterations
→ more stable average

← Previous lab

Measure training time (time.perf_counter)

Next lab →

Regression metrics — MAE, MSE, RMSE, R²