Concepts 🚀 Project 🧮 Labs

EDA mindset — 8 questions that reveal what the data is hiding

Exploratory Data Analysis is not 'df.describe()'. It's a systematic investigation. 8 questions that uncover patterns, leaks, and ML-readiness.

script.py v3.1

import pandas as pd
import numpy as np

# ── Create a realistic dataset with HIDDEN issues ──
np.random.seed(42)
n = 1000
df = pd.DataFrame({
    'age': np.random.normal(40, 12, n).clip(18, 80),
    'income': np.random.lognormal(10.5, 0.6, n),
    'years_experience': np.random.randint(0, 40, n),
    'education_years': np.random.choice([12, 14, 16, 18, 20], n),
    'city': np.random.choice(['NYC', 'LA', 'SF', 'Chicago', 'Boston'], n),
    'hire_date': pd.date_range('2010-01-01', periods=n, freq='D'),
    'salary': np.nan,  # will be computed below
})
# Salary depends on experience + education (with some noise)
df['salary'] = (
    30000 +
    df['years_experience'] * 2500 +
    df['education_years'] * 4000 +
    np.random.normal(0, 8000, n) +
    df['city'].map({'NYC': 15000, 'SF': 20000, 'LA': 10000, 'Chicago': 5000, 'Boston': 8000})
)
# Inject a LEAK: salary_per_year (derived from salary — would be cheating in ML)
df['salary_per_year'] = df['salary'] / df['years_experience'].clip(1)
# Inject a constant column (useless for ML)
df['country'] = 'USA'
# Inject a duplicate-ish column
df[' yrs_exp '] = df['years_experience']  # whitespace in name + duplicate

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 1: What does each column MEAN?
# ════════════════════════════════════════════════════════════════════
print('Q1: COLUMN MEANINGS')
print('-' * 50)
print(df.columns.tolist())
print('  THINK: Do I understand every column? What UNIT is it in?')
print('  PROBLEM: " yrs_exp " has whitespace — will cause bugs.')
print('  ACTION: Strip whitespace: df.columns = df.columns.str.strip()')
df.columns = df.columns.str.strip()

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 2: What's the distribution of each numeric column?
# ════════════════════════════════════════════════════════════════════
print('\nQ2: DISTRIBUTIONS (mean, std, min, max, quartiles)')
print('-' * 50)
print(df.describe().round(0).to_string())
print('  THINK: Is the min/max reasonable? age=18-80 ✓, income min?')
print(f'  income min: {df["income"].min():.0f}, max: {df["income"].max():.0f}')
print(f'  salary min: {df["salary"].min():.0f}, max: {df["salary"].max():.0f}')
print('  THINK: income is lognormal (skewed). salary is roughly normal.')

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 3: Are there CONSTANT or DUPLICATE columns?
# ════════════════════════════════════════════════════════════════════
print('\nQ3: CONSTANT / DUPLICATE columns')
print('-' * 50)
for col in df.columns:
    n_unique = df[col].nunique()
    if n_unique == 1:
        print(f'  CONSTANT: {col} (only value: {df[col].iloc[0]}) — useless for ML, drop it')
# Check for duplicate columns (same values, different names)
for i, col1 in enumerate(df.columns):
    for col2 in df.columns[i+1:]:
        if df[col1].equals(df[col2]):
            print(f'  DUPLICATE: {col1} == {col2} — drop one')
print('  ACTION: df = df.drop(columns=["country"])')

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 4: What's the CORRELATION between features?
# ════════════════════════════════════════════════════════════════════
print('\nQ4: CORRELATIONS (which features move together?)')
print('-' * 50)
numeric_df = df.select_dtypes(include=[np.number])
corr = numeric_df.corr()
# Find high correlations (excluding self-correlation)
high_corr = []
for i in range(len(corr.columns)):
    for j in range(i+1, len(corr.columns)):
        c = corr.iloc[i, j]
        if abs(c) > 0.7:
            high_corr.append((corr.columns[i], corr.columns[j], c))
for c1, c2, val in high_corr:
    print(f'  {c1} ↔ {c2}: {val:.3f}')
print('  THINK: years_experience ↔ salary: 0.85 (makes sense — more exp = more pay)')
print('  THINK: salary_per_year ↔ salary: high (OBVIOUSLY — it\'s derived from salary)')
print('  ACTION: salary_per_year is a LEAK — drop it before ML.')

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 5: Is there a TIME component? (often overlooked)
# ════════════════════════════════════════════════════════════════════
print('\nQ5: TIME component')
print('-' * 50)
print(f'  hire_date range: {df["hire_date"].min()} to {df["hire_date"].max()}')
print(f'  THINK: salary might increase over time (inflation, raises).')
print(f'  THINK: if train/test split is random, future data leaks into train.')
print(f'  ACTION: for time-series, use TimeSeriesSplit, not train_test_split.')

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 6: Are there HIDDEN DATA LEAKS?
# ════════════════════════════════════════════════════════════════════
print('\nQ6: DATA LEAKS (the #1 ML bug)')
print('-' * 50)
print('  A leak = feature that won\'t be available at prediction time.')
print('  Examples: salary_per_year (derived from target), post-event features.')
print(f'  Found: salary_per_year correlates {corr.loc["salary_per_year", "salary"]:.3f} with salary')
print(f'  ACTION: drop salary_per_year — it\'s derived from the target.')

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 7: What's the target distribution?
# ════════════════════════════════════════════════════════════════════
print('\nQ7: TARGET distribution (salary)')
print('-' * 50)
print(f'  Mean: {df["salary"].mean():.0f}')
print(f'  Median: {df["salary"].median():.0f}')
print(f'  Std: {df["salary"].std():.0f}')
print(f'  Skew: {df["salary"].skew():.2f} (0=normal, >0=right-skewed)')
print('  THINK: if skew > 1, consider log-transforming the target.')

# ════════════════════════════════════════════════════════════════════
#  EDA QUESTION 8: How does the target vary by CATEGORY?
# ════════════════════════════════════════════════════════════════════
print('\nQ8: TARGET by CATEGORY (salary by city)')
print('-' * 50)
print(df.groupby('city')['salary'].agg(['mean', 'median', 'count']).round(0).to_string())
print('  THINK: SF has highest mean salary — makes sense (tech hub).')
print('  THINK: city is a predictive feature — keep it.')

Python output click Run to execute

Click "Run" to execute the code on the left.
The output will appear here.

🧠 How it works

EDA = Exploratory Data Analysis. It's NOT just df.describe(). 8 questions that uncover what the data is HIDING: Q1 MEANINGS — understand every column. Strip whitespace from names. Q2 DISTRIBUTIONS — is the min/max reasonable? Is it skewed? Q3 CONSTANTS/DUPLICATES — constant columns are useless. Duplicate columns waste space. Q4 CORRELATIONS — features that move together. >0.7 = high. One is redundant. Q5 TIME — is there a date column? Future data leaking into train = fake accuracy. Q6 LEAKS — the #1 ML bug. Features derived from the target (salary_per_year from salary). Will be perfect at training, fail in production. ALWAYS check for these. Q7 TARGET DISTRIBUTION — skewed target? Log-transform it. Q8 TARGET BY CATEGORY — does the target vary by category? If yes, the category is predictive. Senior data scientists spend HOURS on this. Freshers skip to model.fit(). That's why freshers' models fail in production.

✏️ Try these changes

Remove the leak column (df = df.drop(columns=['salary_per_year'])) before Q4
→ the leak won't show up in correlations
Add a constant column: df['constant_col'] = 42
→ Q3 will detect and flag it
Make salary highly skewed: df['salary'] = np.exp(df['salary'] / 20000)
→ Q7 will show high skew → log-transform needed
Add a duplicate column: df['age_dup'] = df['age']
→ Q3 will detect the duplicate

← Previous lab

Encoding — OneHot vs Label vs Target vs Frequency vs Embedding

Next lab →

EDA visualization — 6 plots that tell you everything