🚀 Stage 3 of 13

Clean the data + split into X and y

Drop nulls, encode categoricals, separate features (X) from target (y). The last cleaning step before ML.

🧠 The intuition sklearn models refuse to train on dirty data. 'Dirty' means: nulls, non-numeric columns, and the target column mixed into the features. Cleaning fixes all three. - nulls -> fill with median (numeric) or mode (categorical) - categoricals -> one-hot encode (turn 'low/med/high' into 3 binary columns) - X / y split -> X = everything except final_score, y = final_score After this step, X is a 2D numpy array of numbers, y is a 1D array of numbers. That's the only format sklearn accepts.

✏️ Tiny task — write this yourself Split the DataFrame into X and y. Just 2 lines: X = df.drop(columns=['final_score', 'student_id']) y = df['final_score'] Then check X.shape and y.shape. X should be (1000, 6) and y should be (1000,).

✅ Check — what you should see X.shape == (1000, 6) # 1000 students, 6 features y.shape == (1000,) # 1000 final scores If X still has final_score in it, your model will cheat by reading the answer. If X still has student_id, the model will memorize IDs instead of learning patterns.

🧮 Code files for this stage

📄 clean.py python

import pandas as pd

df = pd.read_csv('data/students.csv')

# Step 1: drop the ID column — it's not a feature
df = df.drop(columns=['student_id'])

# Step 2: check nulls
print(df.isnull().sum())
# If any column has nulls, fill them:
#   df = df.fillna(df.median(numeric_only=True))

# Step 3: encode categoricals (parental_support is already numeric 1-5)
# If you had a text column like 'internet_quality' = low/med/high:
#   df = pd.get_dummies(df, columns=['internet_quality'], drop_first=True)

# Step 4: split into X (features) and y (target)
X = df.drop(columns=['final_score'])
y = df['final_score']

print('X shape:', X.shape)   # (1000, 6)
print('y shape:', y.shape)   # (1000,)
print('Features:', list(X.columns))

How it works

drop(student_id) is critical — IDs are unique per row, so the model would just memorize 'student 1 -> 68' instead of learning the pattern 'more study hours -> higher score'. drop(final_score) is even more critical — leaving the target in X is called 'data leakage' and gives you 100% accuracy that completely fails on new students. get_dummies turns a categorical column into N binary columns — essential for sklearn which only accepts numbers.

🚧 Routes — what's wired up at this stage Still no routes. Cleaning is offline. The cleaned data lives in memory (or you can save it as X.csv and y.csv if you want to inspect).

🧮 Is a response model necessary here? NO — use response_class=HTMLResponse

This is sklearn/pandas, not FastAPI. No response model needed.

⚠️ Common mistakes to avoid

Forgetting to drop student_id — model memorizes instead of generalizing.
Leaking the target into X — model reports 99% accuracy and fails on real students.
Calling get_dummies BEFORE the train/test split — causes 'feature mismatch' at prediction time when a new student has a category the model never saw.

👀 What's next Next: Stage 4 trains your first ML model — LinearRegression — with fit() and score(). Just 4 lines.

Next — Stage 4

Your first ML model — 4 lines that do the magic

→