Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.
0%
Analysis
Layers-
Neurons-
Total Params-
Selection
Hover over nodes for details.
Right-click to edit | Scroll to Zoom
Overview
What dropout does
On each training forward pass, each neuron in a dropout layer is on its own set to zero with chance p. A other random subset is dropped each step. So the network never trains the same design twice.
At test time time dropout is switched off fully and all neurons are used. That asymmetry is the point. And the source of the most common bug, since a net left in training mode gives other predictions each time it is called. The scaling can be done at test time (multiply outputs by 1 - p) or at training time (divide surviving outputs by 1 - p, called "inverted dropout"). PyTorch and TensorFlow both use inverted dropout. So no scaling is needed at test time. But the net must still be in eval() mode. Or the random zeroing continues and predictions are non-deterministic. Dropout rates between 0.2 and 0.5 are standard: more rates force more redundancy but slow convergence. For convolutional layers, spatial dropout (dropping whole feature maps rather than individual pixels) works better since neighbouring pixels are correlated and dropping one pixel has little effect when its neighbours survive. Dropout is less common in modern designs that use batch normalisation. It gives its own smoothing through mini-batch noise. The two can be mixed. But the mix is subtle: batch norm's stats shift when dropout changes the output spread. So the order matters. This page trains a small network with and without dropout, plots the training and validation loss curves side by side. And shows how dropout closes the overfitting gap.
Randomly switch off a fraction of neurons on every training step. It sounds destructive and it is one of the most effective regularisers ever found, because it stops neurons relying on each other.
What dropout does
On each training forward pass, each neuron in a dropout layer is on its own set to zero with chance p. A other random subset is dropped each step. So the network never trains the same design twice.
At test time time dropout is switched off fully and all neurons are used. That asymmetry is the point. And the source of the most common bug, since a net left in training mode gives other predictions each time it is called. The scaling can be done at test time (multiply outputs by 1 - p) or at training time (divide surviving outputs by 1 - p, called "inverted dropout"). PyTorch and TensorFlow both use inverted dropout. So no scaling is needed at test time. But the net must still be in eval() mode. Or the random zeroing continues and predictions are non-deterministic. Dropout rates between 0.2 and 0.5 are standard: more rates force more redundancy but slow convergence. For convolutional layers, spatial dropout (dropping whole feature maps rather than individual pixels) works better since neighbouring pixels are correlated and dropping one pixel has little effect when its neighbours survive. Dropout is less common in modern designs that use batch normalisation. It gives its own smoothing through mini-batch noise. The two can be mixed. But the mix is subtle: batch norm's stats shift when dropout changes the output spread. So the order matters. This page trains a small network with and without dropout, plots the training and validation loss curves side by side. And shows how dropout closes the overfitting gap.
Why removing capacity improves transfer
The mechanism is the prevention of co-adaptation. Without dropout, neurons specialise in tandem: neuron A learns to correct neuron B’s systematic error. It just works while B behaves just as it did in training. That partnership is a form of memorisation and it does not survive new data.
With dropout, B may be absent on any given step. So A cannot depend on it. Each neuron is forced to learn a feature that is useful on its own, against a randomly varying set of colleagues. The result is redundant, reliable forms rather than brittle chains.
The second reading is that dropout approximates an ensemble. A layer of n units has 2n possible dropout masks. So training samples from an exponential family of thinned networks sharing one set of weights. With all units at test time approximates averaging their predictions. Ensembles transfer better than single nets. And this is an ensemble that costs nothing extra to train.
The scaling detail
Dropping units changes the expected magnitude of a layer’s output. If half the inputs are zeroed, the sum arriving at the next layer is about half its usual size. So a network trained with dropout would see inputs twice as large at test time, when nothing is dropped.
Frameworks fix this with inverted dropout: in training, surviving outputs are divided by (1 − p). With p = 0.5 the survivors are doubled, restoring the expected sum. And test time needs no adjustment at all. This is automatic in PyTorch and TensorFlow. And worth knowing about since it explains why a hand-rolled dropout build often makes a net worse.
Randomly switching units off
Dropout is a smoother with an unusual mechanism: in each training step, each unit in a layer is set to zero with chance p, on its own.
With p = 0.5, roughly half the layer disappears on each batch. A other half each time.
Two ways to read why that helps:
It stops co-adaptation. Without dropout, units can form fragile committees where one unit's output just makes sense given another's. Since any unit may vanish, each has to give something on its own useful.
It is an implicit ensemble. A network with n droppable units sets 2ⁿ. Possible sub-networks, all sharing weights. Training samples from them. Test time approximates averaging over them. Ensembles reduce variance. And this is a very cheap way to get one.
The empirical effect is real and large: dropout was one of the main reasons deep networks became trainable on limited data in the early 2010s.
Train and test time differ
At training time, units are dropped and the left outputs are scaled up by 1/(1−p). "inverted dropout" — so that the expected total stays the same.
At test time time, nothing is dropped and no scaling is applied. The layer turns a no-op.
That is why the mode switch matters:
model.train() # dropout active
# ... training ...
model.eval() # dropout disabled
with torch.no_grad():
preds = model(x)
Forgetting net.eval() gives predictions that differ from run to run for the same input. It is confusing to debug and easy to miss if you just look at aggregate metrics.
How much, and where
Location
Typical rate
After dense layers
0.3–0.5
Between convolutional layers
0.0–0.1, or none
Transformer sublayers
0.1
Input layer
0.1–0.2, rarely used
Output layer
Never
The convolutional row surprises people. Standard dropout works poorly between convolutions, since neighbouring pixels in a feature map are highly correlated. Dropping individual outputs removes little information. DropBlock and SpatialDropout, which drop contiguous regions or whole channels, work much better when smoothing is needed there.
In modern convolutional designs, dropout has largely been replaced by batch normalisation, weight decay and heavy data augmentation. It stays standard in transformers and in the dense heads of older designs.
Tuning it is straightforward: raise p if training score far exceeds validation score, less it if both are stuck low and training is slow to converge.
Break the network on purpose, then put it back
Dropout zeroes random units in training and scales the survivors so the expected output does not change. That scaling is the part that gets built wrong.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
a = np.ones((4, 8))
p = 0.5
print("a layer's output, 4 rows of 8 units, all 1.0:")
mask = (rng.random(a.shape) > p)
dropped = a * mask
print(" keep mask (p_drop=%.1f):" % p)
print(mask.astype(int))
print(" after dropping, before any scaling -- row sums:", dropped.sum(axis=1))
print(" the next layer expected around %d. it is now getting about %d."
% (a.shape[1], dropped.sum(axis=1).mean()))
print()
print("INVERTED dropout, which is what every framework actually does:")
scaled = dropped / (1 - p)
print(" divide the survivors by (1 - p) = %.1f" % (1 - p))
print(" row sums now:", scaled.sum(axis=1))
print(" over many draws the expectation is preserved:")
sums = []
for _ in range(20000):
m = (rng.random(8) > p)
sums.append((np.ones(8) * m / (1 - p)).sum())
print(" mean row sum over 20,000 draws: %.4f (target %d)"
% (np.mean(sums), 8))
print()
print("the scaling happens at TRAINING time, which is why inference needs no")
print("special case: you simply stop dropping and change nothing else.")
print("the original paper scaled at test time instead, and that is the")
print("version that leads to 'my model is fine in training and broken in")
print("production' -- you have to remember to do something.")
print()
Output
The next block continues with the left demonstrations.
example_02.pyNumPy
import numpy as np
print("what it costs in variance. one row, same weights, 10 forward passes:")
w = rng.normal(0, 0.5, 8)
h = rng.normal(1.0, 0.3, 8)
outs = []
for i in range(10):
m = (rng.random(8) > p)
outs.append(((h * m / (1 - p)) @ w))
print(" outputs:", np.round(outs, 4))
print(" no dropout:", round(h @ w, 4))
print(" mean %.4f, sd %.4f -- the prediction is now a random variable."
% (np.mean(outs), np.std(outs)))
print()
print("the rate matters, and 0.5 is not a default so much as a starting point:")
for pd in (0.0, 0.1, 0.3, 0.5, 0.8, 0.95):
trials = [((h * (rng.random(8) > pd) / max(1 - pd, 1e-9)) @ w) for _ in range(4000)]
print(" p=%.2f expected units kept %.1f/8 output sd %.4f"
% (pd, 8 * (1 - pd), np.std(trials)))
print(" at p=0.95 almost nothing survives and the surviving unit is")
print(" multiplied by 20. the signal is gone and the noise is enormous.")
print()
print("why it regularises, in one sentence: a unit cannot rely on any")
print("particular other unit being present, so the network cannot build")
print("fragile co-adapted chains. here is that co-adaptation being broken:")
print(" two units that always fire together can share the work 50/50.")
print(" with dropout, each must carry the load alone half the time, so")
print(" both end up individually useful rather than jointly useful.")
print()
Output
The next block continues with the left demonstrations.
example_03.pyNumPy
import numpy as np
print("where to put it, and where not to:")
print(" after a dense layer's activation -- the standard place")
print(" NOT between a conv layer and its BatchNorm, where it corrupts the")
print(" statistics BatchNorm is trying to estimate")
print(" rarely alongside BatchNorm at all -- BN already injects batch noise,")
print(" and the two together often underperform either alone")
print(" transformers use it on attention weights and after each sublayer,")
print(" usually at 0.1, not 0.5")
print()
print("and the one that catches people: at eval time dropout must be OFF.")
print("if your validation loss is mysteriously higher than training loss on")
print("the same data, this is the first thing to check.")
Output
Tests to try
Turn it off. Set Dropout to 0 and press Simulate Flow. Each neuron participates and signal reaches the output along each path.
Use the standard setting. Set Dropout to 50 and simulate repeatedly. A other half is silenced each time. The network is a other shape on each pass. It is just what breaks co-adaptation.
Push it too far. Set Dropout to 90. So little signal survives that some paths carry nothing at all. Too much dropout does not regularise, it starves the network. Visible here as an underfitting failure.
Change the width. Raise Input and simulate at 50% again. A wider layer keeps more absolute signal at the same rate. It is why large layers tolerate more dropout than small ones.
Common mistakes
Leaving the net in training mode at test time. Forgetting net.eval() in PyTorch means dropout stays active and predictions turn random. This is the single most common dropout bug. And it also silently affects batch normalisation.
Applying it to a net that is underfitting. Dropout costs capacity. If training loss is now high, dropout makes both training and validation worse.
With p = 0.5 all over. That figure comes from large fully connected layers. Convolutional layers often want 0.1–0.2, since their weight sharing is now a strong smoother. And input layers want very little.
Mixing it carelessly with batch normalisation. The two interact badly. Dropout changes the variance batch norm estimated. Common practice is batch norm instead of dropout in convolutional nets. Or dropout just after the normalisation layer.
Key takeaway
Dropout zeroes a random fraction of units on each training pass and disables it at test time. It forces each neuron to be on its own useful rather than co-adapted to set partners. Read as an ensemble, it trains exponentially many thinned networks that share weights and averages them for free. It regularises a net that is overfitting and harms one that is not. And the scaling that makes it work is handled by the framework. Provided you remember to switch the net into test mode.
Where dropout sits among the regularisers
Method
What it does
Best for
Dropout
Randomly zeroes activations
Dense layers, transformers
Weight decay (L2)
Penalises large weights
Everything; always on
Early stopping
Stops when validation stops improving
Everything; free
Data augmentation
Expands the effective dataset
Images, audio, text
Batch normalisation
Normalises activations, adds batch noise
CNNs
Label smoothing
Softens the targets
Classification
Mixup / CutMix
Blends examples and labels
Images
These mix, and the sensible default stack for a small dataset is weight decay plus early stopping plus augmentation, with dropout added if a gap stays.
Note that dropout and batch normalisation interact awkwardly. Batch normalisation now injects noise through batch stats. And the variance shift dropout adds can conflict with the running stats. If you use both, put dropout after the normalisation. And expect to need a less rate than you would without it.
Dropout at test time: measuring uncertainty
There is one deliberate use of dropout at prediction time. Keep it active, run the same input twenty times. And the predictions vary since a other sub-network answers each time.
The spread of those predictions is an estimate of the net's uncertainty — Monte Carlo dropout. It is not a rigorous Bayesian posterior. But it is cheap, needs no retraining. And is far better than the raw softmax confidence for flagging inputs the net has no basis for judging.
model.train() # keep dropout on deliberately
with torch.no_grad():
preds = torch.stack([model(x) for _ in range(20)])
mean, std = preds.mean(0), preds.std(0) # prediction and uncertainty
Use net.train() carefully here — it also puts batch normalisation into training mode. It is often not what you want. Enabling just the dropout modules is the correct way in a real build.
Questions people ask
What dropout rate should I use? 0.5 after dense layers is the classic starting point. 0.1 in transformers. Tune from the size of the train/validation gap.
Should I use dropout with batch normalisation? Often unneeded. If both, put dropout after normalisation and use a less rate.
Why do my predictions change between runs?net.eval() was not called. So dropout is still active.
Does dropout slow training? It slows convergence in epochs, since each step trains a other sub-network. Wall-clock cost per step is negligible.
Is dropout still used? Yes in transformers, less in modern convolutional networks where augmentation and normalisation do the work.
Can I use dropout on the input? You can, at a low rate. And it behaves like adding noise to the data. Augmentation is often a better use of the same idea.
Recap in one screen
Dropout zeroes a random fraction of units each training step, stopping units from relying on each other.
It approximates training an ensemble of exponentially many sub-networks that share weights.
Active in training, disabled at test time — call net.eval().
0.5 for dense layers, 0.1 for transformers. And rarely between convolutions.
Modern vision nets substitute augmentation and batch normalisation. Keeping dropout on at test time gives a cheap uncertainty estimate.
Check yourself
0 of 3
Answer without scrolling back up.
What does dropout do during training?
A different random subset is silenced every pass, so no unit can rely on any particular other unit being present. That forced redundancy is what regularises the network.
Is dropout active at inference time?
You want deterministic predictions when serving. Leaving dropout on at inference is a classic bug: it makes the same input return different answers on each call.
A dropout rate of 0.9 on a small network is likely to cause:
Regularisation is a dial, not a switch. Drop nine units in ten and there is barely a network left to learn anything.
Cheat sheet
Dropout Regularization
Switch neurons off at random during training and watch dropout stop the network leaning on any single pathway.
DEEP LEARNING · vizlearn.in/deep_learning/dropout_in_neural_networks.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
Dropout is a regularization technique used to prevent overfitting in neural networks.
During training, randomly selected neurons are ignored (dropped out). They are temporarily removed from the network, meaning they make no contribution to the activation of downstream neurons on the forward pass, and weight updates are not applied to the neuron on the backward pass.
Use the slider to simulate dropout percentage. Notice how connections disappear as neurons are deactivated, forcing the network to learn more robust features that don't rely on specific neurons.