Standard 2 stops to get here

Standardization

Rescaling each feature to mean 0 and standard deviation 1, so features measured in different units are on a comparable scale.

Your route here

2 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  3. Standardization · you are here ✓ understood

Standardization rescales a feature so that its values have a mean of 0 and a standard deviation of 1. For each value, subtract the feature’s mean and divide by its standard deviation. The result, often called a z-score, says how many standard deviations a value sits above or below average. Scikit-learn’s StandardScaler does exactly this.

Why bother

Raw features arrive in whatever units they were recorded in. Income might run into the hundreds of thousands while age tops out near 100. Many methods are sensitive to that. A distance-based method or an RBF-kernel SVM will be dominated by the feature with the biggest numbers. Penalties like L2 regularization treat all weights alike, which only makes sense if the features share a scale. And gradient descent tends to converge faster when inputs are centred near zero and have similar variances, a point LeCun and colleagues made in their 1998 guide to training neural networks.

Tree-based models are the notable exception: a decision tree splits on thresholds, so rescaling a feature doesn’t change what it learns.

Standardization vs. normalization

The two words are often used loosely. Strictly, standardization centres and scales by the standard deviation, while min-max normalization squeezes values into a fixed range such as 0 to 1. Standardization has no fixed range, and extreme outliers still distort the mean and standard deviation it relies on, so robust scalers based on the median are used for heavy-tailed data.

The common mistake

Compute the mean and standard deviation on the training set only, then apply those same numbers to the validation and test sets. Computing them on the full dataset first lets information about the test data into training, a quiet form of data leakage.

Where it sits

Before this

Feature
Standardization

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Foundations Normalization Scaling features to a standard range (typically 0-1 using min-max scaling) to improve model training and convergence. Often used interchangeably with standardization (mean=0, std=1), though technically distinct. Foundations Data Preprocessing Cleaning, transforming, and preparing raw data for model training (handling missing values, normalization, encoding). Neural Networks Batch Normalization A technique that normalizes layer inputs to stabilize and accelerate training by reducing internal covariate shift. Foundations Feature Engineering The process of selecting, transforming, and creating input features to improve model performance. Evaluation Data Leakage When information from outside the training data is used to create the model, leading to overly optimistic performance estimates. Training Gradient Descent An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

Sources

  1. scikit-learn User Guide, Preprocessing data: Standardization, or mean removal and variance scaling . Official documentation, section 8.3.1
  2. Yann LeCun, Léon Bottou, Genevieve B. Orr and Klaus-Robert Müller, "Efficient BackProp" . Neural Networks: Tricks of the Trade, Springer, 1998; section 4.3, Normalizing the Inputs