Adjust the sliders below to pass different values. Notice how raw, unscaled inputs (like Salary = 80,000) cause massive color shifts in the connections, leading to an imbalanced network. Use the scaler to normalize the data and stabilize training.
Overview
Why scale breaks gradient descent
The gradient of the loss with respect to a weight is proportional to that weight’s input. So a feature measured in hundreds of thousands produces gradients roughly four orders of magnitude larger than a feature measured in tens.
That leaves no workable learning rate. Set it small enough to keep the salary weight stable and the age weight barely moves; set it large enough for age and salary diverges. The loss surface is a long narrow valley, and gradient descent bounces across it instead of running down it.
Scaling makes the valley round. With comparable input ranges the gradients are comparable, one learning rate suits every weight, and convergence takes a fraction of the steps.
35
75k
Network Analysis
Network Status
STABLE
Weights are updating evenly.
Current Input Data Passed
Age (Node 1)
Raw: 00.0
Salary (Node 2)
Raw: 00.0
Avg Weight Magnitudes
Age Weights (W1)0.100
Salary Weights (W2)0.100
Drag to Pan | Scroll to Zoom
Feature Scaling & Weight Bias: A Practical Guide
Age runs from 18 to 80; salary runs from 20,000 to 200,000. Feed both into a network unscaled and the salary weight receives gradients thousands of times larger - so no single learning rate can train both.
Standardisation and normalisation
Standardisation (z-score) centres on zero with unit variance:
x′ = (x − μ) / σ
Output is unbounded but typically within −3 to +3. It preserves the shape of the distribution and handles outliers without letting them compress everything else. This is the sensible default for neural networks.
Min-max normalisation maps to a fixed range, usually [0, 1]:
x′ = (x − min) / (max − min)
Useful when a bounded input is required, and fragile: a single extreme outlier sets the maximum and squashes every ordinary value into a narrow band near zero.
Robust scaling uses the median and interquartile range instead of mean and standard deviation, which is the right choice when outliers are present and genuine.
Work the numbers
Take age 30 and salary 60,000 with weights of 0.5 each. The salary term contributes 30,000 to the weighted sum and the age term contributes 15 — the age feature is invisible, and it would take a weight around 1000× larger to compete.
Standardise first. If age has mean 45 and standard deviation 15, and salary has mean 80,000 with standard deviation 40,000:
age′ = (30 − 45) / 15 = −1.0
salary′ = (60000 − 80000) / 40000 = −0.5
Both are now around 1 in magnitude, both contribute comparably, and both weights receive gradients of a similar size.
Why unscaled inputs break training
A network's first layer computes w · x + b. If one feature ranges over 0–1 and another over 0–200,000, then the second feature dominates that sum entirely, and the gradient with respect to its weight is hundreds of thousands of times larger than the gradient for the first.
Three things go wrong at once.
The loss surface becomes a ravine. Steep in one direction, nearly flat in another. A learning rate small enough to be stable in the steep direction is far too small to make progress in the flat one, so training crawls.
Activations saturate. A weighted sum in the tens of thousands pushes a sigmoid or tanh flat against its asymptote, where the derivative is essentially zero and the unit stops learning.
Initialisation assumptions break. He and Xavier initialisation both assume inputs of roughly unit variance. Feed a variance of 10⁹ and the first layer's activations are enormous from the first forward pass.
The fix is one line, and it is not optional. Unlike tree models, which compare features individually and are indifferent to units, every neural network needs its inputs on a comparable scale.
The three methods
Method
Formula
Result
Use when
Standardisation
(x − μ) / σ
Mean 0, sd 1
The default for networks
Min-max
(x − min) / (max − min)
Range 0–1
Bounded inputs such as pixels
Robust
(x − median) / IQR
Median 0
Outliers present
Standardisation is the usual choice. It is unbounded, so an outlier does not compress everything else into a narrow band, and zero-centred inputs suit the symmetric initialisation schemes.
Min-max guarantees a range, which is why pixel values are divided by 255. Its weakness is sensitivity to outliers: one extreme value squashes every other value towards zero.
Robust scaling uses the median and interquartile range, so a handful of extreme values cannot distort the transformation.
For heavily skewed features — income, counts, populations — a log transform before scaling usually helps more than the choice of scaler, because it addresses the shape rather than just the range.
Fit on training data only
This is the rule that makes the difference between an honest evaluation and a leak.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # compute mu and sigma HERE
X_val = scaler.transform(X_val) # apply the same numbers
X_test = scaler.transform(X_test)
Calling fit_transform on validation or test data computes their means and standard deviations, which means information from those sets has influenced the training pipeline. The reported score is then optimistic, sometimes substantially.
Two further practical points. The fitted scaler must be saved and shipped with the model — a model served with unscaled inputs, or scaled by recomputed statistics, produces confident nonsense. And for neural networks, the target is often scaled too, in which case the predictions must be inverse-transformed before any error is reported, or the RMSE is in the wrong units.
Why an unscaled input slows everything down
Two features on different scales make the loss surface an ellipse, and gradient descent is bad at ellipses. This measures the condition number, then fixes it three ways.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 400
raw = np.column_stack([rng.normal(0.5, 0.02, n), # e.g. a ratio
rng.normal(60_000, 15_000, n)]) # e.g. an income
w_true = np.array([3.0, 0.00004])
y = raw @ w_true + rng.normal(0, 0.05, n)
print("two features:")
print(" feature 0: mean %8.3f, sd %8.3f" % (raw[:, 0].mean(), raw[:, 0].std()))
print(" feature 1: mean %8.1f, sd %8.1f" % (raw[:, 1].mean(), raw[:, 1].std()))
print()
def design(a):
# every variant gets a bias column, or a centred feature set could not
# represent y's mean and the comparison would be meaningless
return np.column_stack([a, np.ones(len(a))])
def cond(a):
M = design(a)
ev = np.linalg.eigvalsh(M.T @ M / len(M))
return ev.max() / max(ev.min(), 1e-30), ev.max()
def standardize(a): return (a - a.mean(axis=0)) / a.std(axis=0)
def minmax(a): return (a - a.min(axis=0)) / (a.max(axis=0) - a.min(axis=0))
def center(a): return a - a.mean(axis=0)
print("%-24s %18s %16s %20s"
% ("preprocessing", "condition number", "max curvature", "max stable lr"))
variants = [("raw", raw), ("centred only", center(raw)),
("min-max to [0,1]", minmax(raw)), ("standardised", standardize(raw))]
for name, A in variants:
c, L = cond(A)
print("%-24s %18.3e %16.3e %20.3e" % (name, c, L, 2 / (2 * L)))
print()
print("the condition number is the ratio of the steepest curvature to the")
print("shallowest. gradient descent needs roughly that many steps to")
print("converge, so a condition number of 1e9 means it will not converge in")
print("any budget you have.")
print()
print("run it and see. the same %d steps on each version:" % 400)
for name, A in variants:
c, L = cond(A)
lr = 0.9 * 2 / (2 * L) # just under each version's own limit
M = design(A)
w = np.zeros(M.shape[1])
for _ in range(400):
w -= lr * 2 * M.T @ (M @ w - y) / n
resid = ((M @ w - y) ** 2).mean()
print(" %-22s lr %10.3e final MSE %14s"
% (name, lr,
"%.6f" % resid if np.isfinite(resid) and resid < 1e9 else "diverged"))
print()
best = ((design(standardize(raw)) @ np.linalg.lstsq(design(standardize(raw)),
y, rcond=None)[0] - y) ** 2).mean()
print(" the exact least-squares answer, for reference: MSE %.6f" % best)
print()
print("every version uses the largest learning rate that is stable FOR IT,")
print("so this is not a handicap -- it is the best each one can do in 400")
print("steps. the two scaled versions reach the exact optimum; the other two")
print("do not get close.")
print()
print("note that 'centred only' came out WORSE than raw. subtracting the mean")
print("changes where the surface sits without changing how elongated it is,")
print("so the condition number stays astronomical -- and it happened to")
print("remove the one large gradient that was letting the raw version make")
print("any progress at all. centring is not scaling, and on its own it is")
print("not reliably an improvement.")
print()
print("why it happens, geometrically. a step moves each weight by")
print("lr x gradient, and the gradient for a weight is scaled by its input:")
g = 2 * raw.T @ (raw @ np.zeros(2) - y) / n
print(" at w = 0, the two gradient components are:")
print(" feature 0: %14.4f" % g[0])
print(" feature 1: %14.4f" % g[1])
print(" a factor of %.0f apart. one learning rate has to serve both, so it"
% abs(g[1] / g[0]))
print(" is set by the larger one and the smaller weight barely moves.")
print()
print("which scaler to use:")
print(" StandardScaler (mean 0, sd 1) -- the default. works with any")
print(" distribution and is what most initialisation schemes assume.")
print(" MinMaxScaler ([0,1]) -- when you need a bounded range, e.g.")
print(" image pixels. one outlier squashes everything else.")
print(" RobustScaler (median, IQR) -- when outliers are real data.")
print()
print("three rules that are not optional:")
print(" 1. fit the scaler on TRAINING data only. fitting on everything")
print(" leaks the test set's mean into training.")
print(" 2. save the scaler with the model. an unscaled input at serving")
print(" time produces confident nonsense, silently.")
print(" 3. scale the TARGET too for regression -- a target in the millions")
print(" produces gradients in the millions, and the same problem.")
print()
print("and the reason BatchNorm and LayerNorm exist: the inputs to layer 5")
print("drift out of scale during training even if layer 1's inputs were")
print("perfect. scaling the input fixes the first layer. normalisation")
print("layers fix the rest of them, continuously.")
Output
Guided experiments
Train unscaled. Set Data Normalization Method to none, put Salary Input near 200000 and Age Input near 30, and press Start Training. Training is unstable or crawls, because one input dominates every gradient.
Standardise and repeat. Switch Data Normalization Method to standardisation, press Reset Weights and train again with the same inputs. Convergence is faster and far smoother.
Compare the scalers. Run min-max against standardisation on the same values. Min-max compresses everything into [0, 1]; standardisation centres on zero and keeps the spread — which is why it pairs better with activations symmetric about zero.
Change the learning rate under each. With scaling off, raise Learning Rate and watch it diverge. With scaling on, the same rate is stable. Scaling widens the range of learning rates that work at all.
Where this goes wrong
Fitting the scaler on the full dataset. Computing the mean and standard deviation before splitting leaks test information into training. Fit on train, then transform validation and test with those same statistics.
Refitting the scaler at inference. Production data must use the training statistics. Fitting a new scaler on a single incoming batch produces silently wrong inputs.
Scaling one-hot columns. They are already on a comparable scale, and standardising them destroys sparsity by making every zero non-zero.
Min-max with outliers. One extreme value compresses everything else into a sliver of the range. Use standardisation or robust scaling.
Scaling the target without inverting it. If you scale y for training, predictions come back in scaled units and must be transformed back before they mean anything.
In one line
Gradients scale with input magnitude, so unscaled features give some weights enormous gradients and others negligible ones, and no single learning rate serves both. Standardisation is the default for neural networks; min-max suits bounded inputs and breaks on outliers. Fit the scaler on the training split only, keep those statistics for validation, test and production, and leave one-hot columns alone.
Which models care
Model
Needs scaling?
Why
Neural networks
Yes
Gradients, initialisation, saturation
KNN, k-means, SVM
Yes
Built on distances
Linear/logistic with regularisation
Yes
The penalty is on raw coefficient size
PCA
Yes
It maximises variance, which depends on units
Decision trees, forests, boosting
No
Splits compare one feature at a time
The last row is the useful contrast. A tree asks "is income above 40,000?", and the answer does not change if income is measured in thousands. That is why gradient boosting needs almost no preprocessing and a neural network needs a pipeline.
Note that scaling does not help underfitting or overfitting directly — it makes optimisation work. A model that will not train at all often starts training the moment the inputs are scaled, which is why it is the first thing to check.
Scaling different kinds of input
Images. Divide by 255, then standardise per channel with the dataset's statistics — for ImageNet-pretrained models, the published mean and standard deviation. Getting this wrong costs several points of accuracy with no visible symptom.
Tabular data. Standardise numeric columns; leave one-hot columns alone (they are already 0/1, and scaling them distorts the regularisation penalty).
Text. Not applicable — tokens become embedding lookups, and the embeddings are learned at a sensible scale.
Audio. Normalise per clip, or use log-mel spectrograms, which compress the dynamic range in the same spirit as a log transform.
Time series. Scale using statistics from the training period only — using the whole series leaks future information into the past, which is the same mistake as fitting on the test set.
Questions people ask
Is scaling needed if I use batch normalisation? Yes. Batch normalisation normalises activations between layers; the first layer still receives the raw input.
Standardisation or min-max? Standardisation for most network inputs; min-max for naturally bounded values such as pixels.
Should I scale one-hot columns? No. They are already on a 0–1 scale, and scaling them changes how regularisation treats them.
What about the target variable? Scale it for neural network regression, and invert before reporting. Not needed for tree models.
What happens if a feature has zero variance? Standardisation divides by zero. Drop the column — a constant feature carries no information anyway.
Can I scale after splitting into batches? The statistics must come from the whole training set, not from a batch. Fit once, apply everywhere.
Recap in one screen
Networks need inputs on a comparable scale, or the loss surface becomes a ravine and activations saturate.
Standardise by default; min-max for bounded inputs; robust scaling when outliers are present.
Log-transform heavily skewed features before scaling.
Fit the scaler on training data only, and ship it with the model.
Trees do not need it; every distance-based, regularised or gradient-trained model does.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Gradients scale with input magnitude, so unscaled features give some weights enormous gradients and others negligible ones, and no single learning rate serves both. Standardisation is the default for neural networks; min-max suits bounded inputs and breaks on outliers. Fit the scaler on the training split only, keep those statistics for validation, test and production, and leave one-hot columns alone.
What does this module say about “Why scale breaks gradient descent”?
The gradient of the loss with respect to a weight is proportional to that weight’s input . So a feature measured in hundreds of thousands produces gradients roughly four orders of magnitude larger than a feature measured in tens.
What does this module say about “Work the numbers”?
Take age 30 and salary 60,000 with weights of 0.5 each. The salary term contributes 30,000 to the weighted sum and the age term contributes 15 — the age feature is invisible, and it would take a weight around 1000× larger to compete.
Cheat sheet
Feature Scaling & Weight Bias
The gradient of the loss with respect to a weight is proportional to that weight’s input. So a feature measured in hundreds of thousands produces gradients roughly four orders of magnitude larger than a feature measured in tens.
DEEP LEARNING · vizlearn.in/deep_learning/feature_scaling_in_neural_networks.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.