Visualize the Rectified Linear Unit (ReLU). See how it introduces non-linearity by passing positive signals and turning off (clipping) negative signals.
ReLU Function Graph
PROCESSING...
Mathematical Formula
A = max(0, Z)
Click "Forward Pass" to view calculations.
Overview
What is an Output Function?
When a Convolutional or Dense layer processes data, it computes a series of linear math-based steps (dot products). If a neural network just consisted of linear steps, stacking multiple layers wouldn't help—the whole network would in math collapse into a single linear net. To learn complex, real-world patterns (like recognizing a face or a dog), we must add non-linearity. This is the job of the Output Function. ReLU (Rectified Linear Unit) is the default choice for hidden layers: it outputs the input unchanged if positive. And zero if negative — f(x) = max(0, x). That one line makes the network nonlinear. And its simplicity is just why it works so well: it is fast to compute, fast to differentiate (the gradient is 1 for positive inputs and 0 for negative). And it does not saturate for large positive vals the way sigmoid and tanh do. Saturation causes vanishing gradients since the gradient ways zero. It slows training to a crawl in deep networks. ReLU's weakness is the "dying ReLU" problem: a neuron whose input is always negative has zero gradient and stops learning permanently. Leaky ReLU fixes this by letting a small slope (often 0.01) for negative inputs. And variants like ELU and GELU offer smoother transitions. This page applies ReLU to a feature map, shows which vals survive and which are zeroed. And covers the dying-ReLU problem with a worked case.
The simple mathematical trick that powers modern Deep Learning.
What is an Output Function?
When a Convolutional or Dense layer processes data, it computes a series of linear math-based steps (dot products). If a neural network just consisted of linear steps, stacking multiple layers wouldn't help—the whole network would in math collapse into a single linear net. To learn complex, real-world patterns (like recognizing a face or a dog), we must add non-linearity. This is the job of the Output Function. ReLU (Rectified Linear Unit) is the default choice for hidden layers: it outputs the input unchanged if positive. And zero if negative — f(x) = max(0, x). That one line makes the network nonlinear. And its simplicity is just why it works so well: it is fast to compute, fast to differentiate (the gradient is 1 for positive inputs and 0 for negative). And it does not saturate for large positive vals the way sigmoid and tanh do. Saturation causes vanishing gradients since the gradient ways zero. It slows training to a crawl in deep networks. ReLU's weakness is the "dying ReLU" problem: a neuron whose input is always negative has zero gradient and stops learning permanently. Leaky ReLU fixes this by letting a small slope (often 0.01) for negative inputs. And variants like ELU and GELU offer smoother transitions. This page applies ReLU to a feature map, shows which vals survive and which are zeroed. And covers the dying-ReLU problem with a worked case.
Enter ReLU (Rectified Linear Unit)
ReLU is now the most popular output function in the world for hidden layers. Its formula is brilliantly simple: if the input is negative, output 0. If the input is positive, output the input unchanged.
In math: f(x) = max(0, x)
In a biological sense, think of a node as a neuron. If the incoming signal (pre-output) is negative, the neuron decides it's not relevant and does not "fire" (outputs 0). If the signal is positive, it fires proportionately to the signal's strength.
The simplest useful non-linearity
ReLU(x) = max(0, x)
Negative vals turn zero. Positive vals pass through unchanged. That is the whole function. And it replaced far more sophisticated options since of what it does to training.
Its derivative is equally simple: just 1 for positive inputs, just 0 for negative ones.
That derivative of 1 is the key property. The chain rule multiplies one factor per layer. So a network of sigmoids. Whose derivative peaks at 0.25. Loses a factor of at least four per layer. And after ten layers the gradient reaching the input is a millionth of what left the loss. ReLU adds a factor of just 1 wherever it is active. So gradients pass through depth intact.
This is not a small optimisation. Networks deeper than a handful of layers were not practically trainable before ReLU. And much of the deep learning era rests on this one-line function.
Why any non-linearity is needed at all
Without one, stacked layers collapse. Two linear layers compute W₂(W₁x) = (W₂W₁)x. It is a single matrix. So a hundred linear layers can express just what one can: a straight edge.
Inserting a non-linear function between layers is what lets a network represent curves, corners and disjoint regions. Depth just buys expressiveness when something bends between the layers.
ReLU bends in the simplest possible way. One kink at the origin — and that turns out to be enough. A network of ReLUs is a piecewise-linear function. And with enough pieces it can approximate anything.
The other properties that matter
Speed. A comparison against zero, against an exponential for sigmoid or tanh. On billions of outputs that gap is real. And it applies to both the forward and backward pass.
Sparsity. Roughly half the outputs in a randomly set up layer are zero. And trained networks often reach 50–80% sparsity. Sparse outputs are cheaper. And there is evidence they make forms more disentangled.
No saturation on the positive side. Sigmoid flattens at both ends. So a strongly activated neuron learns almost nothing. ReLU's positive side never flattens.
The price is the negative side. A neuron whose inputs always give a negative pre-output outputs zero, gets a gradient of zero. And stops updating — permanently. This is the dying ReLU problem. And a network can lose a large fraction of its neurons this way, often after a too-large learning rate has pushed weights into a bad region.
Guided Tests with This Interactive
Observe the Clipping:
Look at the Pre-Output sliders on the left. Set a few to negative counts and a few to positive. Notice the Outputs on the right at once clip the negative ones to just 0.00 (turning grey), while the positive ones carry over perfectly.
Watch the Forward Pass:
Click the Forward Pass button. Watch the animation on the center canvas. You will see just where each input lands on the $y = \max(0, x)$ curve. Notice that any point on the left side of the Y-axis drops straight to the bottom floor line.
Find Sparsity:
Click the "Randomize Vals" button a few times. On average, about half of the outputs will be zero. This creates a "sparse" network where just a subset of neurons is active at any given time. This makes the network highly fast and helps stop neurons from turning overly dependent on each other.
The "Dying ReLU" Problem
Since ReLU outputs just 0 for any negative input, its gradient (slope) for negative vals is also 0. In backpropagation, if a neuron always gets negative inputs, its weights will never update. It turns a "dead" neuron.
To solve this, variants like Leaky ReLU were invented. It let a tiny, non-zero gradient for negative counts (e.g., $f(x) = 0.01x$ when $x < 0$).
What ReLU does to a feature map
ReLU is one line -- max(x, 0) -- and it is the just reason a deep stack of convolutions is worth building. This shows what it does to real feature-map vals, why the option is not a deeper network but a pointless one. And the failure mode it brings with it.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(5)
# a small image with a bright bar, and two edge kernels
img = np.full((7, 12), 50.0)
img[:, 5:8] = 200.0
img += rng.normal(0, 5, (7, 12))
K_left = np.array([[-1., 0, 1]] * 3) / 3.0 # responds + to a dark->light edge
K_right = -K_left # the mirror image
def conv(a, k):
p = np.pad(a, ((1, 1), (1, 1)), mode="edge")
return np.array([[float((p[i:i + 3, j:j + 3] * k).sum())
for j in range(a.shape[1])] for i in range(a.shape[0])])
print("done")
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
fm = conv(img, K_left)
print("A FEATURE MAP from one edge kernel, row 3:")
print(" " + " ".join("%6.1f" % v for v in fm[3]))
print(" positive at the LEFT edge of the bar (dark to light) and negative")
print(" at the right edge (light to dark). same kernel, opposite signs.")
print()
print("AFTER ReLU -- max(x, 0):")
print(" " + " ".join("%6.1f" % v for v in np.maximum(fm[3], 0)))
print(" the right edge is gone. not attenuated -- gone, exactly zero.")
print(" this feature now means 'dark-to-light edge here', a claim that")
print(" is either made or not made. before ReLU it meant 'dark-to-light")
print(" by this much, OR light-to-dark by this much', which is two")
print(" different things sharing one number.")
print()
print(" the network's answer to losing the other edge is not to recover")
print(" it -- it is to learn a SECOND kernel that is the mirror image:")
fm2 = conv(img, K_right)
print(" mirrored kernel + ReLU, row 3:")
print(" " + " ".join("%6.1f" % v for v in np.maximum(fm2[3], 0)))
print(" this is why filter counts are what they are. a linear layer needs")
print(" one filter per direction of variation; a ReLU layer needs one per")
print(" DIRECTED half of it, so a ReLU network needs roughly twice the")
print(" channels to represent the same variation -- a real cost, paid")
print(" because the resulting features are far easier to build on.")
print()
print("SPARSITY -- how much of a feature map survives:")
print("%-30s %12s %12s" % ("", "nonzero", "fraction"))
for label, a in (("before ReLU", fm), ("after ReLU", np.maximum(fm, 0))):
nz = int((np.abs(a) > 1e-9).sum())
print("%-30s %12d %11.1f%%" % (label, nz, 100.0 * nz / a.size))
print()
Output
The next block covers the left steps.
example_03.pyNumPy
import numpy as np
print("NOW THE POINT THAT MATTERS. WITHOUT a nonlinearity, stacking layers")
print("buys you nothing at all. two convolutions in a row, no activation:")
A = rng.normal(0, 1, (3, 3))
B = rng.normal(0, 1, (3, 3))
x = rng.normal(0, 1, (9, 9))
two_layers = conv(conv(x, A), B)
# the composition of two convolutions is a single convolution with the
# kernels convolved together -- build that 5x5 kernel explicitly
C = np.zeros((5, 5))
for i in range(3):
for j in range(3):
C[i:i + 3, j:j + 3] += B[i, j] * A
def conv5(a, k):
p = np.pad(a, ((2, 2), (2, 2)), mode="edge")
return np.array([[float((p[i:i + 5, j:j + 5] * k).sum())
for j in range(a.shape[1])] for i in range(a.shape[0])])
Output
The next block continues with the left demonstrations.
example_04.pyNumPy
import numpy as np
one_layer = conv5(x, C)
inner = slice(3, 6)
print(" two 3x3 convolutions, interior values:")
print(" " + " ".join("%8.4f" % v for v in two_layers[4][inner]))
print(" ONE 5x5 convolution with the combined kernel:")
print(" " + " ".join("%8.4f" % v for v in one_layer[4][inner]))
print(" max difference in the interior: %.2e"
% np.abs(two_layers[2:-2, 2:-2] - one_layer[2:-2, 2:-2]).max())
print(" identical. a hundred stacked linear layers collapse into ONE")
print(" linear layer, exactly, always. depth without a nonlinearity is")
print(" not a deep model -- it is an expensive way to write a shallow one.")
print()
print(" now put a ReLU between them and try to collapse it:")
relu_stack = conv(np.maximum(conv(x, A), 0), B)
print(" with ReLU: " + " ".join("%8.4f" % v for v in relu_stack[4][inner]))
print(" difference from the single 5x5: %.4f"
% np.abs(relu_stack[2:-2, 2:-2] - one_layer[2:-2, 2:-2]).max())
print(" no 5x5 kernel reproduces it, because the function is no longer")
print(" linear. THAT is what the activation buys: it makes depth mean")
print(" something.")
print()
print("WHY ReLU AND NOT A SMOOTH CURVE. compare gradients:")
print("%-16s %14s %14s %14s" % ("input x", "sigmoid'(x)", "tanh'(x)", "ReLU'(x)"))
Output
Building on the same setup, the next block shows the left cases.
example_05.pyNumPy
import numpy as np
for v in (-6.0, -2.0, 0.0, 2.0, 6.0):
sg = 1 / (1 + np.exp(-v))
print("%-16.1f %14.6f %14.6f %14.0f"
% (v, sg * (1 - sg), 1 - np.tanh(v) ** 2, 1.0 if v > 0 else 0.0))
print(" ReLU is not differentiable at exactly 0; every framework simply")
print(" picks 0 there, and it has never mattered in practice.")
print(" sigmoid's gradient at x=6 is %.6f. multiply ten of those together"
% (1 / (1 + np.exp(-6.0)) * (1 - 1 / (1 + np.exp(-6.0)))))
print(" for a ten-layer network and the gradient reaching layer 1 is")
print(" %.2e -- the vanishing gradient, and it is why nothing deep"
% (1 / (1 + np.exp(-6.0)) * (1 - 1 / (1 + np.exp(-6.0)))) ** 10)
print(" trained before ReLU. ReLU's gradient is exactly 1 wherever the")
print(" unit is active, so it passes gradient through unchanged, however")
print(" many layers deep.")
print()
print("AND THE FAILURE IT INTRODUCES -- DEAD UNITS. ReLU's gradient is")
print("exactly 0 on the other side. a unit whose bias drifts far negative")
print("outputs 0 for every input, so it gets 0 gradient, so it can never")
print("recover:")
z = rng.normal(0, 1, 2000)
print("%-24s %14s %14s" % ("bias", "fraction active", "gradient flow"))
for bias in (0.5, 0.0, -1.0, -3.0, -6.0):
act = (z + bias) > 0
print("%-24.1f %14.3f %14s"
% (bias, act.mean(), "dead" if act.mean() == 0 else "ok"))
print(" at bias -6 not one of %d inputs activates the unit. it is a" % z.size)
print(" constant 0 that will stay a constant 0 forever, and it still")
print(" costs memory and multiplications on every forward pass.")
print(" the fixes all keep a little slope on the left: LeakyReLU uses")
print(" 0.01x, ELU and GELU curve smoothly through the origin. all of")
print(" them trade a little of ReLU's exact sparsity for the guarantee")
print(" that no unit is ever permanently unreachable.")
Output
Worth remembering
ReLU stands for Rectified Linear Unit.
It replaces negative vals with 0 and keeps positive vals unchanged.
It gives key non-linearity to CNNs.
It is computationally very cheap compared to functions like Sigmoid or Tanh.
It promotes network sparsity (many inactive nodes).
The variants, and when they help
Activation
Negative side
Notes
ReLU
Exactly 0
The default; fast; can die
Leaky ReLU
0.01x
Keeps a small gradient alive
PReLU
αx, with α learned
Slightly better, slightly more to fit
ELU
α(eˣ − 1)
Smooth, mean closer to zero, more expensive
GELU
x · Φ(x)
Smooth; standard in transformers
SiLU / Swish
x · sigmoid(x)
Smooth; common in modern vision models
For convolutional networks, plain ReLU stays the sensible default. Leaky ReLU is the first thing to try if you observe many dead units. GELU and SiLU dominate in transformers and in recent fast vision designs. There their smoothness appears to help slightly — at a small computational cost.
Stopping dead units matters more than choosing among the variants:
Use He setup, which is designed for ReLU and scales the initial weights by √(2/fan_in). Xavier setup, designed for tanh, leaves ReLU networks with outputs that shrink through depth.
Less the learning rate if you see a sudden collapse in the fraction of active units.
Add batch normalisation before the output. It keeps pre-outputs centred and makes the dead region much harder to fall into.
Where ReLU sits in a block
The conventional order in a convolutional network is:
Conv → BatchNorm → ReLU
Batch normalisation before the output is the arrangement the first paper used and it stays standard. It keeps the input to ReLU centred so that roughly half the units are active. It is what the setup assumed.
Note that the convolution's bias is redundant when batch normalisation follows, since the normalisation subtracts the mean anyway. Which is why bias=False is common on convolutions in ResNet-style code.
In residual blocks the placement is more delicate: the pre-output variant (BN → ReLU → Conv) trains deeper networks better, since it leaves the identity path fully clean. And that detail was worth a paper of its own.
Questions people ask
How do I know if my ReLUs are dying? Log the fraction of zero outputs per layer. Around 50% is healthy. 90%+ that keeps rising is a problem.
Is ReLU differentiable at zero? Not strictly — there is a corner. Frameworks set the derivative there as 0 by convention. And it causes no real trouble since just zero almost never occurs.
Should I use ReLU on the output layer? Just for regression targets that must be non-negative. Sorting uses softmax or sigmoid. Unbounded regression uses no output.
Does ReLU cause exploding gradients? It does not damp them, since the positive-side derivative is 1. Mixed with poor setup or a high learning rate that can let explosion. It is what gradient clipping and normalisation address.
Why does ReLU work despite discarding half the information? Since the network compensates by learning filters in both polarities. And the sparsity it gives appears to be useful rather than merely lossy.
Is GELU better than ReLU? Marginally, in transformers. In convolutional networks the gap is often within noise. And ReLU is faster.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.