vizlearn
Home / Deep Learning

Dropout in Neural Networks

Deep LearningBy Updated

Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.

Overview

What dropout does

On each training forward pass, each neuron in a dropout layer is independently set to zero with probability p. A different random subset is dropped every step, so the network never trains the same architecture twice.

At inference time dropout is switched off entirely and all neurons are used. That asymmetry is the point — and the source of the most common bug, since a model left in training mode gives different predictions every time it is called.

0%

Analysis

Layers -
Neurons -
Total Params -

Selection

Hover over nodes for details.
Right-click to edit | Scroll to Zoom

Dropout Regularization: A Practical Guide

Randomly switch off a fraction of neurons on every training step. It sounds destructive and it is one of the most effective regularisers ever found, because it stops neurons relying on each other.

Why removing capacity improves generalisation

The mechanism is the prevention of co-adaptation. Without dropout, neurons specialise in tandem: neuron A learns to correct neuron B’s systematic error, which only works while B behaves exactly as it did in training. That partnership is a form of memorisation and it does not survive new data.

With dropout, B may be absent on any given step, so A cannot depend on it. Every neuron is forced to learn a feature that is useful on its own, against a randomly varying set of colleagues. The result is redundant, robust representations rather than brittle chains.

The second reading is that dropout approximates an ensemble. A layer of n units has 2n possible dropout masks, so training samples from an exponential family of thinned networks sharing one set of weights; using all units at inference approximates averaging their predictions. Ensembles generalise better than single models, and this is an ensemble that costs nothing extra to train.

The scaling detail

Dropping units changes the expected magnitude of a layer’s output. If half the inputs are zeroed, the sum arriving at the next layer is about half its usual size — so a network trained with dropout would see inputs twice as large at inference, when nothing is dropped.

Frameworks fix this with inverted dropout: during training, surviving activations are divided by (1 − p). With p = 0.5 the survivors are doubled, restoring the expected sum, and inference needs no adjustment at all. This is automatic in PyTorch and TensorFlow, and worth knowing about because it explains why a hand-rolled dropout implementation often makes a model worse.

Randomly switching units off

Dropout is a regulariser with an unusual mechanism: during each training step, every unit in a layer is set to zero with probability p, independently.

With p = 0.5, roughly half the layer disappears on every batch — a different half each time.

Two ways to read why that helps:

It prevents co-adaptation. Without dropout, units can form fragile committees where one unit's output only makes sense given another's. Since any unit may vanish, each has to produce something independently useful.

It is an implicit ensemble. A network with n droppable units defines 2ⁿ possible sub-networks, all sharing weights. Training samples from them; inference approximates averaging over them. Ensembles reduce variance, and this is a very cheap way to get one.

The empirical effect is real and large: dropout was one of the main reasons deep networks became trainable on limited data in the early 2010s.

Train and inference differ

At training time, units are dropped and the remaining activations are scaled up by 1/(1−p) — "inverted dropout" — so that the expected total stays the same.

At inference time, nothing is dropped and no scaling is applied. The layer becomes a no-op.

That is why the mode switch matters:

model.train()      # dropout active
# ... training ...

model.eval()       # dropout disabled
with torch.no_grad():
    preds = model(x)

Forgetting model.eval() produces predictions that differ from run to run for the same input, which is confusing to debug and easy to miss if you only look at aggregate metrics.

How much, and where

LocationTypical rate
After dense layers0.3–0.5
Between convolutional layers0.0–0.1, or none
Transformer sublayers0.1
Input layer0.1–0.2, rarely used
Output layerNever

The convolutional row surprises people. Standard dropout works poorly between convolutions, because neighbouring pixels in a feature map are highly correlated — dropping individual activations removes little information. DropBlock and SpatialDropout, which drop contiguous regions or entire channels, work much better when regularisation is needed there.

In modern convolutional architectures, dropout has largely been replaced by batch normalisation, weight decay and heavy data augmentation. It remains standard in transformers and in the dense heads of older designs.

Tuning it is straightforward: raise p if training accuracy far exceeds validation accuracy, lower it if both are stuck low and training is slow to converge.

Break the network on purpose, then put it back

Dropout zeroes random units during training and scales the survivors so the expected output does not change. That scaling is the part that gets implemented wrong.

example_01.pyNumPy
Output

                

                

                    
Experiments to try

                    
  1. Turn it off. Set Dropout to 0 and press Simulate Flow. Every neuron participates and signal reaches the output along every path.
  2. Use the standard setting. Set Dropout to 50 and simulate repeatedly. A different half is silenced each time — the network is a different shape on every pass, which is exactly what breaks co-adaptation.
  3. Push it too far. Set Dropout to 90. So little signal survives that some paths carry nothing at all. Too much dropout does not regularise, it starves the network — visible here as an underfitting failure.
  4. Change the width. Raise Input and simulate at 50% again. A wider layer keeps more absolute signal at the same rate, which is why large layers tolerate higher dropout than small ones.

                

                

                    
Common mistakes

                    
  • Leaving the model in training mode at inference. Forgetting model.eval() in PyTorch means dropout stays active and predictions become random. This is the single most common dropout bug, and it also silently affects batch normalisation.
  • Applying it to a model that is underfitting. Dropout costs capacity. If training loss is already high, dropout makes both training and validation worse.
  • Using p = 0.5 everywhere. That figure comes from large fully connected layers. Convolutional layers usually want 0.1–0.2, since their weight sharing is already a strong regulariser, and input layers want very little.
  • Combining it carelessly with batch normalisation. The two interact badly — dropout changes the variance batch norm estimated. Common practice is batch norm instead of dropout in convolutional nets, or dropout only after the normalisation layer.

                

                

                    
Key takeaway

                    
Dropout zeroes a random fraction of units on every training pass and disables itself at inference, which forces each neuron to be independently useful rather than co-adapted to specific partners. Read as an ensemble, it trains exponentially many thinned networks that share weights and averages them for free. It regularises a model that is overfitting and harms one that is not, and the scaling that makes it work is handled by the framework — provided you remember to switch the model into evaluation mode.

                

                

                    
Where dropout sits among the regularisers

                    
MethodWhat it doesBest for
DropoutRandomly zeroes activationsDense layers, transformers
Weight decay (L2)Penalises large weightsEverything; always on
Early stoppingStops when validation stops improvingEverything; free
Data augmentationExpands the effective datasetImages, audio, text
Batch normalisationNormalises activations, adds batch noiseCNNs
Label smoothingSoftens the targetsClassification
Mixup / CutMixBlends examples and labelsImages
These combine, and the sensible default stack for a small dataset is weight decay plus early stopping plus augmentation, with dropout added if a gap remains.
Note that dropout and batch normalisation interact awkwardly. Batch normalisation already injects noise through batch statistics, and the variance shift dropout introduces can conflict with the running statistics. If you use both, put dropout after the normalisation, and expect to need a lower rate than you would without it.

                

                

                    
Dropout at inference: measuring uncertainty

                    
There is one deliberate use of dropout at prediction time. Keep it active, run the same input twenty times, and the predictions vary because a different sub-network answers each time.
The spread of those predictions is an estimate of the model's uncertainty — Monte Carlo dropout. It is not a rigorous Bayesian posterior, but it is cheap, requires no retraining, and is far better than the raw softmax confidence for flagging inputs the model has no basis for judging.
model.train()                       # keep dropout on deliberately
with torch.no_grad():
    preds = torch.stack([model(x) for _ in range(20)])
mean, std = preds.mean(0), preds.std(0)     # prediction and uncertainty
Use model.train() carefully here — it also puts batch normalisation into training mode, which is usually not what you want. Enabling only the dropout modules is the correct approach in a real implementation.

                

                

                    
Questions people ask

                    
What dropout rate should I use? 0.5 after dense layers is the classic starting point; 0.1 in transformers. Tune from the size of the train/validation gap.
Should I use dropout with batch normalisation? Often unnecessary. If both, put dropout after normalisation and use a lower rate.
Why do my predictions change between runs? model.eval() was not called, so dropout is still active.
Does dropout slow training? It slows convergence in epochs, because each step trains a different sub-network. Wall-clock cost per step is negligible.
Is dropout still used? Yes in transformers, less in modern convolutional networks where augmentation and normalisation do the work.
Can I use dropout on the input? You can, at a low rate, and it behaves like adding noise to the data. Augmentation is usually a better use of the same idea.

                

                

                    
Recap in one screen

                    
  • Dropout zeroes a random fraction of units each training step, preventing units from relying on each other.
  • It approximates training an ensemble of exponentially many sub-networks that share weights.
  • Active in training, disabled at inference — call model.eval().
  • 0.5 for dense layers, 0.1 for transformers, and rarely between convolutions.
  • Modern vision models substitute augmentation and batch normalisation; keeping dropout on at inference gives a cheap uncertainty estimate.

                

            


        

    




        
    



Check yourself
0 of 3
Answer without scrolling back up.
  1. What does dropout do during training?
  2. Is dropout active at inference time?
  3. A dropout rate of 0.9 on a small network is likely to cause:





 Cheat sheet
Dropout Regularization
On each training forward pass, each neuron in a dropout layer is independently set to zero with probability p. A different random subset is dropped every step, so the network never trains the same architecture twice.
DEEP LEARNING · vizlearn.in/deep_learning/dropout_in_neural_networks.html
More in Deep Learning
Deep LearningRegularization TechniquesDeep LearningEarly StoppingDeep LearningHyperparameter TuningDeep LearningData Sparsity
Further reading
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact