🚀 Stage 7 of 13

Try a better model — RandomForest

Same X, same y, different model. RandomForest captures non-linear patterns that LinearRegression can't.

🧠 The intuition LinearRegression assumes final_score is a weighted sum of features: score = 5 + 2*study_hours + 0.3*attendance + ... That's a plane in 6-dimensional space. RandomForest doesn't assume any shape. It builds 100 decision trees, each one asking yes/no questions like 'study_hours > 5?' and averaging their predictions. It can learn 'students who study > 5 hours AND have attendance > 80 score high, BUT students who study > 5 hours with attendance < 50 score low' — interactions that LinearRegression can't represent. The pattern in real ML: start with LinearRegression (simple, interpretable), then try RandomForest (more accurate, less interpretable), and compare.

✏️ Tiny task — write this yourself Train a RandomForest on the same X, y and compare test R^2. from sklearn.ensemble import RandomForestRegressor rf = RandomForestRegressor(n_estimators=100, random_state=42) rf.fit(X_train, y_train) print(f'Linear R^2: {model.score(X_test, y_test):.3f}') print(f'Forest R^2: {rf.score(X_test, y_test):.3f}')

✅ Check — what you should see RandomForest usually beats LinearRegression by 5-10% R^2 on this kind of data. If RF is WORSE, your data is mostly linear — and LinearRegression is the right choice (simpler, faster, more interpretable).

🧮 Code files for this stage

📄 app/ml/compare_models.py python

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score
import joblib

df = pd.read_csv('data/students.csv').drop(columns=['student_id'])
X = df.drop(columns=['final_score'])
y = df['final_score']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Model A: LinearRegression
lr = LinearRegression().fit(X_train, y_train)
lr_pred = lr.predict(X_test)
print(f'LinearRegression:  R^2={r2_score(y_test, lr_pred):.3f}  MAE={mean_absolute_error(y_test, lr_pred):.2f}')

# Model B: RandomForest
rf = RandomForestRegressor(n_estimators=100, random_state=42).fit(X_train, y_train)
rf_pred = rf.predict(X_test)
print(f'RandomForest:      R^2={r2_score(y_test, rf_pred):.3f}  MAE={mean_absolute_error(y_test, rf_pred):.2f}')

# Save the winner
if r2_score(y_test, rf_pred) > r2_score(y_test, lr_pred):
    joblib.dump(rf, 'app/ml/model.pkl')
    print('Saved RandomForest -> app/ml/model.pkl')
else:
    joblib.dump(lr, 'app/ml/model.pkl')
    print('Saved LinearRegression -> app/ml/model.pkl')

How it works

n_estimators=100 means 'build 100 decision trees and average their predictions'. More trees = more accurate but slower. 100 is the sweet spot for small datasets. random_state=42 makes the result reproducible — without it, every run gives a different forest. We save the winner — the FastAPI app doesn't care which algorithm won, because both LinearRegression and RandomForestRegressor have the same .predict() method. That's the beauty of sklearn's API.

🚧 Routes — what's wired up at this stage Still offline. We're choosing the BEST model before saving it as model.pkl. Once we save it (next stage), the FastAPI app doesn't care which algorithm was used — it just calls .predict().

🧮 Is a response model necessary here? NO — use response_class=HTMLResponse

Model comparison is offline. No response model.

⚠️ Common mistakes to avoid

Using n_estimators=10 — too few trees, model is unstable. Use 100 minimum.
Forgetting random_state — every run gives different scores, you can't compare models fairly.
Picking the model with higher TRAIN R^2 instead of TEST R^2 — you'll pick the overfit one.

👀 What's next Next: Stage 8 is where FastAPI enters. We load model.pkl and expose the FIRST route: POST /predict that takes JSON and returns a score.

Next — Stage 8

First FastAPI route — POST /predict

→