Preparing data

Models are picky about their input. They need numbers, they cannot handle gaps, and many of them work badly when one feature is in the thousands and another is between 0 and 1. Preprocessing turns raw data into the clean numeric matrix a model expects.

Missing values

Options, roughly in order of preference:

1. Understand why the value is missing. Sometimes "missing" is itself a signal worth a column of its own.
2. Fill (impute) numeric gaps with the median, categorical gaps with the most common value.
3. Drop rows only if they are few and missing at random.

import numpy as np
from sklearn.impute import SimpleImputer

X = np.array([[25, 50000], [np.nan, 60000], [35, np.nan], [45, 80000]])
imp = SimpleImputer(strategy="median")
print(imp.fit_transform(X))

Scaling numeric features

Distance based models (k-NN, SVM, k-means), gradient descent and regularised linear models all assume features are on similar scales.

Standardisation subtracts the mean and divides by the standard deviation, so each feature has mean 0 and standard deviation 1:

import numpy as np
from sklearn.preprocessing import StandardScaler

X = np.array([[25, 50000], [30, 60000], [35, 70000], [45, 80000]], dtype=float)
scaler = StandardScaler().fit(X)
print(scaler.transform(X).round(2))
print("means:", scaler.mean_)

Min-max scaling squashes each feature into the range 0 to 1:

import numpy as np
from sklearn.preprocessing import MinMaxScaler

X = np.array([[25, 50000], [30, 60000], [35, 70000], [45, 80000]], dtype=float)
print(MinMaxScaler().fit_transform(X).round(2))

Tree based models do not need scaling, since they only compare a feature with a threshold.

Encoding categories

Text categories must become numbers.

One-hot encoding creates one 0/1 column per category. Use it when the categories have no natural order.

import pandas as pd

df = pd.DataFrame({"city": ["Jaipur", "Delhi", "Pune", "Delhi"], "marks": [88, 72, 60, 79]})
print(pd.get_dummies(df, columns=["city"], dtype=int))

Ordinal encoding maps categories to integers. Use it only when order is meaningful (small < medium < large).

from sklearn.preprocessing import OrdinalEncoder

sizes = [["small"], ["large"], ["medium"], ["small"]]
enc = OrdinalEncoder(categories=[["small", "medium", "large"]])
print(enc.fit_transform(sizes).ravel())

Label encoding the target column of a classifier is fine and scikit-learn does it for you.

Fit on train, transform both

A scaler or imputer learns numbers (means, medians) from data. Those numbers must come from the training set only. If you fit on the whole dataset, information about the test set leaks into training and your evaluation becomes too optimistic.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X = np.arange(20, dtype=float).reshape(10, 2)
X_train, X_test = train_test_split(X, test_size=0.3, random_state=0)

scaler = StandardScaler().fit(X_train)     # learn from train only
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)        # apply the same numbers
print(X_test_s.round(2))

Pipelines

A Pipeline chains preprocessing and a model into one object, so the fit-on-train rule is enforced automatically and the whole thing can be saved and reused.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

X = np.array([[25, 50000], [np.nan, 60000], [35, 70000], [45, 80000], [22, 30000], [50, 90000]])
y = [0, 0, 1, 1, 0, 1]

pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), LogisticRegression())
pipe.fit(X, y)
print(pipe.predict([[40, 75000], [20, 35000]]))

Outliers

Extreme values can dominate a mean or a squared error. Inspect them with describe() and box plots; decide case by case whether they are errors to remove or real values to keep. Clipping to a percentile range is a common compromise:

import numpy as np

x = np.array([12, 15, 14, 13, 400, 16, 11])
lo, hi = np.percentile(x, [5, 95])
print(np.clip(x, lo, hi))

Feature engineering

Creating new features from existing ones is often the single biggest improvement you can make. Ratios (price per square foot), differences (days since last purchase), date parts (day of week), text lengths, and interactions between features all give a model a clearer view of the pattern.

Practice

1. Take a small table with age, salary and city. Impute a missing salary, standardise the numbers and one-hot encode the city.
2. Explain in one sentence why the scaler must not be fit on the test set.
3. Wrap the steps from task 1 and a LogisticRegression into a single pipeline.