Preprocessing mindset — the 7 questions to ask ANY dataset

Before touching ML, ask these 7 questions of ANY dataset. This is how senior data scientists think — not 'load + fit'.

script.py v3.1

import pandas as pd
import numpy as np

# ── Load ANY dataset (here we use a synthetic one so it runs) ──
from sklearn.datasets import make_classification
X, y = make_classification(
    n_samples=1000, n_features=8, n_informative=5,
    n_redundant=2, n_classes=3, weights=[0.7, 0.2, 0.1],
    random_state=42
)
df = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(8)])
df['target'] = y
# Inject some real-world messiness:
df.loc[np.random.RandomState(42).randint(0, 1000, 50), 'feature_0'] = np.nan  # nulls
df.loc[np.random.RandomState(42).randint(0, 1000, 20), 'feature_1'] = 9999   # outliers
df['feature_2'] = df['feature_2'].clip(-2, 2)  # clipped (censored) values
print(f'Shape: {df.shape}')

# ════════════════════════════════════════════════════════════════════
#  THE 7 QUESTIONS — ask these BEFORE any ML
# ════════════════════════════════════════════════════════════════════

# Q1: How big is it? (rows × cols)
print('\n--- Q1: SIZE ---')
print(f'Rows: {len(df):,} | Columns: {df.shape[1]}')
print(f'Memory: {df.memory_usage(deep=True).sum() / 1e6:.1f} MB')
# THINK: <100 rows = too small for ML. >1M rows = need sampling / Dask.

# Q2: What types are the columns? (numeric vs categorical vs datetime)
print('\n--- Q2: TYPES ---')
print(df.dtypes.value_counts())
# THINK: object dtype usually = strings (need encoding). datetime = time features.

# Q3: How many missing values? (and WHERE?)
print('\n--- Q3: MISSING VALUES ---')
nulls = df.isnull().sum()
print(nulls[nulls > 0])
null_pct = (nulls[nulls > 0] / len(df) * 100).round(1)
print(f'Null %: {dict(null_pct)}')
# THINK: >50% null = drop the column. 5-50% = impute. <5% = impute or drop rows.

# Q4: Are there outliers? (values way outside the normal range)
print('\n--- Q4: OUTLIERS (using IQR method) ---')
for col in df.select_dtypes(include=[np.number]).columns:
    if col == 'target': continue
    q1, q3 = df[col].quantile([0.25, 0.75])
    iqr = q3 - q1
    lower, upper = q1 - 1.5*iqr, q3 + 1.5*iqr
    n_outliers = ((df[col] < lower) | (df[col] > upper)).sum()
    if n_outliers > 0:
        print(f'  {col}: {n_outliers} outliers (range: [{df[col].min():.2f}, {df[col].max():.2f}])')
# THINK: outliers might be errors (remove) or real (keep). Domain knowledge matters.

# Q5: Is the target balanced? (classification only)
print('\n--- Q5: TARGET BALANCE ---')
print(df['target'].value_counts(normalize=True).round(3))
# THINK: 90/10 split = imbalanced. Use class_weight, SMOTE, or stratified split.

# Q6: Are there duplicate rows?
print('\n--- Q6: DUPLICATES ---')
print(f'Duplicate rows: {df.duplicated().sum()}')
# THINK: duplicates = data leakage if same row is in train AND test.

# Q7: What do the first 5 rows LOOK like? (the most underrated question)
print('\n--- Q7: HEAD (look at actual values) ---')
print(df.head().to_string())
# THINK: do the values make sense? Is feature_0 an age? A price? A category code?

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

Senior data scientists don't start with model.fit(). They start with QUESTIONS. 7 questions for ANY dataset: Q1 SIZE — <100 rows = too small. >1M = need sampling. Q2 TYPES — object dtype = strings = need encoding. Q3 NULLS — >50% = drop column. 5-50% = impute. <5% = drop rows or impute. Q4 OUTLIERS — IQR method: anything outside [Q1-1.5*IQR, Q3+1.5*IQR]. Domain knowledge decides keep/remove. Q5 BALANCE — 90/10 = imbalanced. Use class_weight, SMOTE, or stratified split. Q6 DUPLICATES — same row in train + test = data leakage = fake accuracy. Q7 HEAD — the most underrated. Look at actual values. Do they make sense? This is the difference between a fresher ('load + fit') and a senior ('understand first').

✏️ Try these changes

Change n_samples from 1000 to 100
→ Q1 shows too-small dataset
Inject 60% nulls in feature_0: df['feature_0'] = df['feature_0'].where(np.random.rand(len(df)) > 0.6)
→ Q3 shows >50% nulls = drop column
Make the target super imbalanced: weights=[0.95, 0.04, 0.01]
→ Q5 shows severe imbalance
Add a categorical column: df['category'] = np.random.choice(['A','B','C'], len(df))
→ Q2 shows object dtype

Next lab →

Imputation — when to use mean vs median vs mode vs KNN vs MICE