Adjust the sliders below to pass different values. Notice how raw, unscaled inputs (like Salary = 80,000) cause massive color shifts in the connections, leading to an imbalanced network. Use the scaler to normalize the data and stabilize training.
35
75k
Network Analysis
Network Status
STABLE
Weights are updating evenly.
Current Input Data Passed
Age (Node 1)
Raw: 00.0
Salary (Node 2)
Raw: 00.0
Avg Weight Magnitudes
Age Weights (W1)0.100
Salary Weights (W2)0.100
Drag to Pan | Scroll to Zoom
Overview
Why scale breaks gradient descent
The gradient of the loss with respect to a weight is proportional to that weight’s input. So a feature measured in hundreds of thousands gives gradients roughly four orders of magnitude more than a feature measured in tens.
That leaves no workable learning rate. Set it small enough to keep the salary weight stable and the age weight barely moves. Set it large enough for age and salary diverges. The loss surface is a long narrow valley. And gradient descent bounces across it instead of running down it.
Scaling makes the valley round. With comparable input ranges the gradients are comparable, one learning rate suits each weight. And convergence takes a fraction of the steps.
This page checks standardisation (z-score), min-max normalisation. And IQR-based scaling side by side, shows when each is right. And covers the most common mistake — fitting the scaler on the full dataset before splitting. It leaks test statistics into training. The code blocks train the same network with and without scaling so the convergence gap is measured. And the visualisation above lets you switch scaling ways and watch the loss surface reshape in real time. It also covers which net families need scaling (each gradient-trained or distance-based net) and which do not (trees, forests, boosting), and explains why batch normalisation does not remove the need to scale the input layer.
Adjust the sliders below to pass different values.
Age runs from 18 to 80; salary runs from 20,000 to 200,000. Feed both into a network unscaled and the salary weight receives gradients thousands of times larger - so no single learning rate can train both.
Why scale breaks gradient descent
The gradient of the loss with respect to a weight is proportional to that weight’s input. So a feature measured in hundreds of thousands gives gradients roughly four orders of magnitude more than a feature measured in tens.
That leaves no workable learning rate. Set it small enough to keep the salary weight stable and the age weight barely moves. Set it large enough for age and salary diverges. The loss surface is a long narrow valley. And gradient descent bounces across it instead of running down it.
Scaling makes the valley round. With comparable input ranges the gradients are comparable, one learning rate suits each weight. And convergence takes a fraction of the steps.
This page checks standardisation (z-score), min-max normalisation. And IQR-based scaling side by side, shows when each is right. And covers the most common mistake — fitting the scaler on the full dataset before splitting. It leaks test statistics into training. The code blocks train the same network with and without scaling so the convergence gap is measured. And the visualisation above lets you switch scaling ways and watch the loss surface reshape in real time. It also covers which net families need scaling (each gradient-trained or distance-based net) and which do not (trees, forests, boosting), and explains why batch normalisation does not remove the need to scale the input layer.
Standardisation and normalisation
Standardisation (z-score) centres on zero with unit variance:
x′ = (x − μ) / σ
Output is unbounded but often within −3 to +3. It preserves the shape of the distribution and handles outliers without letting them compress all else. This is the sensible default for neural networks.
Min-max normalisation maps to a fixed range, often [0, 1]:
x′ = (x − min) / (max − min)
Useful when a bounded input is needed. And fragile: a single extreme outlier sets the max and squashes each ordinary value into a narrow band near zero.
Reliable scaling uses the median and interquartile range instead of mean and standard deviation. It is the right choice when outliers are present and genuine.
Work the counts
Take age 30 and salary 60,000 with weights of 0.5 each. The salary term adds 30,000 to the weighted sum and the age term adds 15 — the age feature is invisible. And it would take a weight around 1000× more to compete.
Standardise first. If age has mean 45 and standard deviation 15. And salary has mean 80,000 with standard deviation 40,000:
age′ = (30 − 45) / 15 = −1.0
salary′ = (60000 − 80000) / 40000 = −0.5
Both are now around 1 in magnitude, both add comparably. And both weights get gradients of a close size.
Why unscaled inputs break training
A network's first layer computes w · x + b. If one feature ranges over 0–1 and another over 0–200,000, then the second feature dominates that sum fully. And the gradient with respect to its weight is hundreds of thousands of times more than the gradient for the first.
Three things go wrong at once.
The loss surface turns a ravine. Steep in one way, nearly flat in another. A learning rate small enough to be stable in the steep way is far too small to make progress in the flat one. So training crawls.
Activations saturate. A weighted sum in the tens of thousands pushes a sigmoid or tanh flat against its asymptote. There the derivative is in effect zero and the unit stops learning.
Initialisation assumptions break. He and Xavier initialisation both assume inputs of roughly unit variance. Feed a variance of 10⁹ and the first layer's activations are enormous from the first forward pass.
The fix is one line, and it is not optional. Unlike tree nets, which check features individually and are indifferent to units, each neural network needs its inputs on a comparable scale.
The three ways
Method
Formula
Result
Use when
Standardisation
(x − μ) / σ
Mean 0, sd 1
The default for networks
Min-max
(x − min) / (max − min)
Range 0–1
Bounded inputs such as pixels
Reliable
(x − median) / IQR
Median 0
Outliers present
Standardisation is the usual choice. It is unbounded, so an outlier does not compress all else into a narrow band. And zero-centred inputs suit the symmetric initialisation schemes.
Min-max guarantees a range, which is why pixel vals are divided by 255. Its weakness is sensitivity to outliers: one extreme value squashes each other value towards zero.
Reliable scaling uses the median and interquartile range. So a handful of extreme vals cannot distort the transformation.
For heavily skewed features — income, counts, populations — a log transform before scaling often helps more than the choice of scaler, since it addresses the shape rather than just the range.
Fit on training data just
This is the rule that makes the gap between an honest test and a leak.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # compute mu and sigma HERE
X_val = scaler.transform(X_val) # apply the same numbers
X_test = scaler.transform(X_test)
Calling fit_transform on validation or test data computes their means and standard deviations. It means information from those sets has influenced the training pipeline. The reported score is then optimistic, sometimes much.
Two more real points. The fitted scaler must be saved and shipped with the net — a net served with unscaled inputs. Or scaled by recomputed statistics, gives confident nonsense. And for neural networks, the target is often scaled too, in which case the predictions must be inverse-transformed before any error is reported. Or the RMSE is in the wrong units.
Why an unscaled input slows all down
Two features on other scales make the loss surface an ellipse. And gradient descent is bad at ellipses. This measures the condition count, then fixes it three ways.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 400
raw = np.column_stack([rng.normal(0.5, 0.02, n), # e.g. a ratio
rng.normal(60_000, 15_000, n)]) # e.g. an income
w_true = np.array([3.0, 0.00004])
y = raw @ w_true + rng.normal(0, 0.05, n)
print("two features:")
print(" feature 0: mean %8.3f, sd %8.3f" % (raw[:, 0].mean(), raw[:, 0].std()))
print(" feature 1: mean %8.1f, sd %8.1f" % (raw[:, 1].mean(), raw[:, 1].std()))
print()
def design(a):
# every variant gets a bias column, or a centred feature set could not
# represent y's mean and the comparison would be meaningless
return np.column_stack([a, np.ones(len(a))])
def cond(a):
M = design(a)
ev = np.linalg.eigvalsh(M.T @ M / len(M))
return ev.max() / max(ev.min(), 1e-30), ev.max()
def standardize(a): return (a - a.mean(axis=0)) / a.std(axis=0)
def minmax(a): return (a - a.min(axis=0)) / (a.max(axis=0) - a.min(axis=0))
def center(a): return a - a.mean(axis=0)
print("%-24s %18s %16s %20s"
% ("preprocessing", "condition number", "max curvature", "max stable lr"))
variants = [("raw", raw), ("centred only", center(raw)),
("min-max to [0,1]", minmax(raw)), ("standardised", standardize(raw))]
for name, A in variants:
c, L = cond(A)
print("%-24s %18.3e %16.3e %20.3e" % (name, c, L, 2 / (2 * L)))
print()
print("the condition number is the ratio of the steepest curvature to the")
print("shallowest. gradient descent needs roughly that many steps to")
print("converge, so a condition number of 1e9 means it will not converge in")
print("any budget you have.")
Output
Now run gradient descent on each variant, each at its own best learning rate. And check convergence.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 400
raw = np.column_stack([rng.normal(0.5, 0.02, n),
rng.normal(60_000, 15_000, n)])
w_true = np.array([3.0, 0.00004])
y = raw @ w_true + rng.normal(0, 0.05, n)
def design(a):
return np.column_stack([a, np.ones(len(a))])
def cond(a):
M = design(a)
ev = np.linalg.eigvalsh(M.T @ M / len(M))
return ev.max() / max(ev.min(), 1e-30), ev.max()
def standardize(a): return (a - a.mean(axis=0)) / a.std(axis=0)
def minmax(a): return (a - a.min(axis=0)) / (a.max(axis=0) - a.min(axis=0))
def center(a): return a - a.mean(axis=0)
variants = [("raw", raw), ("centred only", center(raw)),
("min-max to [0,1]", minmax(raw)), ("standardised", standardize(raw))]
print("run it and see. the same %d steps on each version:" % 400)
for name, A in variants:
c, L = cond(A)
lr = 0.9 * 2 / (2 * L) # just under each version's own limit
M = design(A)
w = np.zeros(M.shape[1])
for _ in range(400):
w -= lr * 2 * M.T @ (M @ w - y) / n
resid = ((M @ w - y) ** 2).mean()
print(" %-22s lr %10.3e final MSE %14s"
% (name, lr,
"%.6f" % resid if np.isfinite(resid) and resid < 1e9 else "diverged"))
print()
best = ((design(standardize(raw)) @ np.linalg.lstsq(design(standardize(raw)),
y, rcond=None)[0] - y) ** 2).mean()
print(" the exact least-squares answer, for reference: MSE %.6f" % best)
print()
print("every version uses the largest learning rate that is stable FOR IT,")
print("so this is not a handicap -- it is the best each one can do in 400")
print("steps. the two scaled versions reach the exact optimum; the other two")
print("do not get close.")
Output
Centring alone came out worse than raw since it changes where the surface sits without changing how elongated it is. The gradient magnitudes show why one learning rate cannot serve both features.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 400
raw = np.column_stack([rng.normal(0.5, 0.02, n),
rng.normal(60_000, 15_000, n)])
w_true = np.array([3.0, 0.00004])
y = raw @ w_true + rng.normal(0, 0.05, n)
print("note that 'centred only' came out WORSE than raw. subtracting the mean")
print("changes where the surface sits without changing how elongated it is,")
print("so the condition number stays astronomical -- and it happened to")
print("remove the one large gradient that was letting the raw version make")
print("any progress at all. centring is not scaling, and on its own it is")
print("not reliably an improvement.")
print()
print("why it happens, geometrically. a step moves each weight by")
print("lr x gradient, and the gradient for a weight is scaled by its input:")
g = 2 * raw.T @ (raw @ np.zeros(2) - y) / n
print(" at w = 0, the two gradient components are:")
print(" feature 0: %14.4f" % g[0])
print(" feature 1: %14.4f" % g[1])
print(" a factor of %.0f apart. one learning rate has to serve both, so it"
% abs(g[1] / g[0]))
print(" is set by the larger one and the smaller weight barely moves.")
Output
These are the real rules for choosing a scaler and avoiding the common mistakes.
example_04.pyNumPy
print("which scaler to use:")
print(" StandardScaler (mean 0, sd 1) -- the default. works with any")
print(" distribution and is what most initialisation schemes assume.")
print(" MinMaxScaler ([0,1]) -- when you need a bounded range, e.g.")
print(" image pixels. one outlier squashes everything else.")
print(" RobustScaler (median, IQR) -- when outliers are real data.")
print()
print("three rules that are not optional:")
print(" 1. fit the scaler on TRAINING data only. fitting on everything")
print(" leaks the test set's mean into training.")
print(" 2. save the scaler with the model. an unscaled input at serving")
print(" time produces confident nonsense, silently.")
print(" 3. scale the TARGET too for regression -- a target in the millions")
print(" produces gradients in the millions, and the same problem.")
print()
print("and the reason BatchNorm and LayerNorm exist: the inputs to layer 5")
print("drift out of scale during training even if layer 1's inputs were")
print("perfect. scaling the input fixes the first layer. normalisation")
print("layers fix the rest of them, continuously.")
Output
Guided experiments
Train unscaled. Set Data Normalization Way to none, put Salary Input near 200000 and Age Input near 30. And press Start Training. Training is unstable or crawls, since one input dominates each gradient.
Standardise and repeat. Switch Data Normalization Way to standardisation, press Reset Weights and train again with the same inputs. Convergence is faster and far smoother.
Check the scalers. Run min-max against standardisation on the same vals. Min-max compresses all into [0, 1]. Standardisation centres on zero and keeps the spread — which is why it pairs better with activations symmetric about zero.
Change the learning rate under each. With scaling off, raise Learning Rate and watch it diverge. With scaling on, the same rate is stable. Scaling widens the range of learning rates that work at all.
Where this goes wrong
Fitting the scaler on the full dataset. Computing the mean and standard deviation before splitting leaks test information into training. Fit on train, then transform validation and test with those same statistics.
Refitting the scaler at inference. Production data must use the training statistics. Fitting a new scaler on a single incoming batch gives silently wrong inputs.
Scaling one-hot columns. They are now on a comparable scale. And standardising them destroys sparsity by making each zero non-zero.
Min-max with outliers. One extreme value compresses all else into a sliver of the range. Use standardisation or reliable scaling.
Scaling the target without inverting it. If you scale y for training, predictions come back in scaled units and must be transformed back before they mean anything.
In one line
Gradients scale with input magnitude. So unscaled features give some weights enormous gradients and others negligible ones. And no single learning rate serves both. Standardisation is the default for neural networks. Min-max suits bounded inputs and breaks on outliers. Fit the scaler on the training split just, keep those statistics for validation, test and production. And leave one-hot columns alone.
Which nets care
Model
Needs scaling?
Why
Neural networks
Yes
Gradients, initialisation, saturation
KNN, k-means, SVM
Yes
Built on distances
Linear/logistic with regularisation
Yes
The penalty is on raw coefficient size
PCA
Yes
It maximises variance, which depends on units
Decision trees, forests, boosting
No
Splits compare one feature at a time
The last row is the useful contrast. A tree asks "is income above 40,000?". And the answer does not change if income is measured in thousands. That is why gradient boosting needs almost no preprocessing and a neural network needs a pipeline.
Note that scaling does not help underfitting or overfitting right — it makes optimisation work. A net that will not train at all often starts training the moment the inputs are scaled. It is why it is the first thing to check.
Scaling other kinds of input
Pics. Divide by 255, then standardise per channel with the dataset's statistics — for ImageNet-pretrained nets, the published mean and standard deviation. Getting this wrong costs a few points of accuracy with no visible symptom.
Tabular data. Standardise numeric columns. Leave one-hot columns alone (they are now 0/1, and scaling them distorts the regularisation cost).
Text. Not applicable — tokens turn embedding lookups. And the embeddings are learned at a sensible scale.
Audio. Normalise per clip, or use log-mel spectrograms. It compress the dynamic range in the same spirit as a log transform.
Time series. Scale with statistics from the training period just — with the whole series leaks future information into the past. It is the same mistake as fitting on the test set.
Questions people ask
Is scaling needed if I use batch normalisation? Yes. Batch normalisation normalises activations between layers. The first layer still gets the raw input.
Standardisation or min-max? Standardisation for most network inputs. Min-max for naturally bounded vals such as pixels.
Should I scale one-hot columns? No. They are now on a 0–1 scale. And scaling them changes how regularisation treats them.
What about the target var? Scale it for neural network regression. And invert before reporting. Not needed for tree nets.
What happens if a feature has zero variance? Standardisation divides by zero. Drop the column — a constant feature carries no information anyway.
Can I scale after splitting into batches? The statistics must come from the whole training set, not from a batch. Fit once, apply all over.
Recap in one screen
Networks need inputs on a comparable scale. Or the loss surface turns a ravine and activations saturate.
Standardise by default. Min-max for bounded inputs. Reliable scaling when outliers are present.
Log-transform heavily skewed features before scaling.
Fit the scaler on training data just. And ship it with the net.
Trees do not need it. Each distance-based, regularised or gradient-trained net does.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Why scale breaks gradient descent”?
What does this module say about “Work the counts”?
Cheat sheet
Feature Scaling & Weight Bias
See what happens when one feature is measured in thousands and another in decimals, and how scaling rebalances the gradients.
DEEP LEARNING · vizlearn.in/deep_learning/feature_scaling_in_neural_networks.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.