🚀 Stage 6 of 13
←
→
← Previous — Stage 5
What does R^2 = 0.80 actually mean?
Find your worst predictions — error analysis
Don't just calculate a metric. Investigate WHERE the model fails. This is what separates real ML from API-calling.
🧠 The intuition
A metric tells you HOW BAD the model is on average. Error analysis tells you WHERE it's bad — and that's 10x more useful.
Steps:
1. Build a results DataFrame with: Actual, Predicted, Error, Abs_Error
2. Sort by Abs_Error descending — the top rows are your worst predictions
3. Look up the original features for those worst rows (X_test.loc[idx])
4. Ask: 'what's unusual about this student?'
This is where ML becomes interesting. You're no longer doing fit/predict/score — you're debugging the model's reasoning.
✏️ Tiny task — write this yourself
Sort the results by Absolute_Error descending and show the top 10.
results = pd.DataFrame({
'Actual': y_test,
'Predicted': y_pred,
})
results['Error'] = results['Actual'] - results['Predicted']
results['Abs_Error'] = results['Error'].abs()
worst10 = results.sort_values(by='Abs_Error', ascending=False).head(10)
print(worst10)
✅ Check — what you should see
You'll see something like:
Actual Predicted Error Abs_Error
220 47.9 86.4 -38.5 38.5
10 98.0 66.7 31.3 31.3
...
These are the students the model gets badly wrong. Next we ask WHY.
🧮 Code files for this stage
📄
app/ml/error_analysis.py
python
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
df = pd.read_csv('data/students.csv').drop(columns=['student_id'])
X = df.drop(columns=['final_score'])
y = df['final_score']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression().fit(X_train, y_train)
y_pred = model.predict(X_test)
# Build the results table
results = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})
results['Error'] = results['Actual'] - results['Predicted']
results['Abs_Error'] = results['Error'].abs()
# Top 10 worst predictions
worst10 = results.sort_values(by='Abs_Error', ascending=False).head(10)
print('=== WORST 10 PREDICTIONS ===')
print(worst10)
# Look up the features for the very worst prediction
worst_idx = worst10.index[0]
print(f'\n=== FEATURES OF WORST PREDICTION (index {worst_idx}) ===')
print(X_test.loc[worst_idx])
# Compare with a student the model got RIGHT
best10 = results.sort_values(by='Abs_Error').head(10)
best_idx = best10.index[0]
print(f'\n=== FEATURES OF BEST PREDICTION (index {best_idx}) ===')
print(X_test.loc[best_idx])
How it works
Sorting by Abs_Error reveals the model's blind spots. The worst row (Actual 47.9, Predicted 86.4) is a student the model thought would score high but actually scored low. Why? Look at their features — maybe they have high study_hours AND low attendance, and the model weighted study_hours too heavily. The best row tells you what the model is GOOD at — probably 'average students with average everything'. Linear models fail on outliers because they assume a linear relationship. A RandomForest (Stage 7) handles outliers better because it bins features.
🚧 Routes — what's wired up at this stage
No routes. Error analysis is offline — it's how you decide whether to retrain with more features, switch to RandomForest, or accept the current model and ship it.
🧮 Is a response model necessary here?
NO — use response_class=HTMLResponse
Pandas DataFrame analysis. No FastAPI. No response model.
⚠️ Common mistakes to avoid
- Computing the metric and moving on — the metric is the START of the analysis, not the end.
- Not looking up the original features — without X_test.loc[idx], the worst-10 list is just numbers with no story.
- Comparing only the worst prediction — always look at worst AND best to see what the model is good vs bad at.
Next — Stage 7
Try a better model — RandomForest