Quick Context
Feature scaling transforms input features to a common range or distribution. Without it, features with large magnitudes (e.g., salary in thousands) dominate the gradient updates while small-scale features (e.g., age 18–80) are effectively ignored. This creates lopsided weight updates that slow or prevent convergence.
1) Why Neural Networks Need Scaling
Neural networks compute weighted sums: z = w₁x₁ + w₂x₂ + b. If x₁ (age) ranges 18–80 but x₂ (salary) ranges 20,000–200,000, then w₂ needs to be ~1000x smaller than w₁ to produce similar contributions. This creates an extremely elongated loss landscape where gradient descent oscillates along one axis and crawls along the other.
With properly scaled features, the loss landscape becomes more spherical, and gradient descent converges much faster.
2) Scaling Methods
- Min-Max Normalization: x' = (x − min) / (max − min). Scales to [0, 1]. Sensitive to outliers.
- Standardization (Z-score): x' = (x − μ) / σ. Centers at 0, std dev = 1. Most common in deep learning.
- Max-Abs Scaling: x' = x / |max|. Scales to [-1, 1]. Preserves sparsity (zeros stay zero).
- Robust Scaling: x' = (x − median) / IQR. Uses median and interquartile range; robust to outliers.
3) Guided Experiments
- Set Age=35, Salary=80k with no scaling — observe the weight imbalance in the visualization.
- Set Salary to extreme (200k) — see how the salary-connected weights dominate.
- Enable feature scaling — all weights should be more balanced and training more stable.
- Compare convergence speed (number of steps to low loss) with and without scaling.
4) Critical Rules
- Fit on training data only — compute μ and σ from the training set. Apply the same transformation to validation and test sets using training statistics.
- Scale after train/test split — scaling before splitting leaks test information into training (data leakage).
- Save the scaler — at inference time you need the exact same μ and σ values.
- Not all features need scaling — binary features (0/1) and already-normalized features can be left as-is.
5) Common Mistakes
- Fitting the scaler on the entire dataset (including test) — this is data leakage.
- Forgetting to scale at inference time — predictions will be wrong.
- Using Min-Max scaling with extreme outliers — one outlier can compress all normal values into a tiny range.
6) Key Takeaways
- Feature scaling makes the loss landscape more uniform, enabling faster and more stable convergence.
- Standardization (Z-score) is the default choice for neural networks.
- Always fit on training data, apply to all splits, and save the scaler for inference.