Visualize the Rectified Linear Unit (ReLU). See how it introduces non-linearity by passing positive signals and turning off (clipping) negative signals.
Overview
What is an Activation Function?
When a Convolutional or Dense layer processes data, it computes a series of linear mathematical operations (dot products). If a neural network only consisted of linear operations, stacking multiple layers wouldn't help—the entire network would mathematically collapse into a single linear model. To learn complex, real-world patterns (like recognizing a face or a dog), we must introduce non-linearity. This is the job of the Activation Function.
ReLU Function Graph
PROCESSING...
Mathematical Formula
A = max(0, Z)
Click "Forward Pass" to view calculations.
Understanding the ReLU Activation Function
The simple mathematical trick that powers modern Deep Learning.
Enter ReLU (Rectified Linear Unit)
ReLU is currently the most popular activation function in the world for hidden layers. Its formula is brilliantly simple: if the input is negative, output 0. If the input is positive, output the input unchanged.
Mathematically: f(x) = max(0, x)
In a biological sense, think of a node as a neuron. If the incoming signal (pre-activation) is negative, the neuron decides it's not relevant and does not "fire" (outputs 0). If the signal is positive, it fires proportionately to the signal's strength.
The simplest useful non-linearity
ReLU(x) = max(0, x)
Negative values become zero; positive values pass through unchanged. That is the entire function, and it replaced far more sophisticated alternatives because of what it does to training.
Its derivative is equally simple: exactly 1 for positive inputs, exactly 0 for negative ones.
That derivative of 1 is the crucial property. The chain rule multiplies one factor per layer, so a network of sigmoids — whose derivative peaks at 0.25 — loses a factor of at least four per layer, and after ten layers the gradient reaching the input is a millionth of what left the loss. ReLU contributes a factor of exactly 1 wherever it is active, so gradients pass through depth intact.
This is not a small optimisation. Networks deeper than a handful of layers were not practically trainable before ReLU, and much of the deep learning era rests on this one-line function.
Why any non-linearity is needed at all
Without one, stacked layers collapse. Two linear layers compute W₂(W₁x) = (W₂W₁)x, which is a single matrix — so a hundred linear layers can express exactly what one can: a straight boundary.
Inserting a non-linear function between layers is what lets a network represent curves, corners and disjoint regions. Depth only buys expressiveness when something bends between the layers.
ReLU bends in the simplest possible way — one kink at the origin — and that turns out to be enough. A network of ReLUs is a piecewise-linear function, and with enough pieces it can approximate anything.
The other properties that matter
Speed. A comparison against zero, against an exponential for sigmoid or tanh. On billions of activations that difference is real, and it applies to both the forward and backward pass.
Sparsity. Roughly half the activations in a randomly initialised layer are zero, and trained networks often reach 50–80% sparsity. Sparse activations are cheaper, and there is evidence they make representations more disentangled.
No saturation on the positive side. Sigmoid flattens at both ends, so a strongly activated neuron learns almost nothing. ReLU's positive side never flattens.
The price is the negative side. A neuron whose inputs always produce a negative pre-activation outputs zero, receives a gradient of zero, and stops updating — permanently. This is the dying ReLU problem, and a network can lose a substantial fraction of its neurons this way, usually after a too-large learning rate has pushed weights into a bad region.
Guided Experiments with This Interactive
Observe the Clipping:
Look at the Pre-Activation sliders on the left. Set a few to negative numbers and a few to positive. Notice the Outputs on the right immediately clip the negative ones to exactly 0.00 (turning grey), while the positive ones carry over perfectly.
Watch the Forward Pass:
Click the Forward Pass button. Watch the animation on the center canvas. You will see exactly where each input lands on the $y = \max(0, x)$ curve. Notice that any point on the left side of the Y-axis drops straight to the bottom floor line.
Identify Sparsity:
Click the "Randomize Values" button a few times. On average, about half of the outputs will be zero. This creates a "sparse" network where only a subset of neurons is active at any given time. This makes the network highly efficient and helps prevent neurons from becoming overly dependent on each other.
The "Dying ReLU" Problem
Because ReLU outputs exactly 0 for any negative input, its gradient (slope) for negative values is also 0. During backpropagation, if a neuron consistently receives negative inputs, its weights will never update. It becomes a "dead" neuron.
To solve this, variants like Leaky ReLU were invented, which allow a tiny, non-zero gradient for negative numbers (e.g., $f(x) = 0.01x$ when $x < 0$).
What ReLU does to a feature map
ReLU is one line -- max(x, 0) -- and it is the only reason a deep stack of convolutions is worth building. This shows what it does to real feature-map values, why the alternative is not a deeper network but a pointless one, and the failure mode it brings with it.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(5)
# a small image with a bright bar, and two edge kernels
img = np.full((7, 12), 50.0)
img[:, 5:8] = 200.0
img += rng.normal(0, 5, (7, 12))
K_left = np.array([[-1., 0, 1]] * 3) / 3.0 # responds + to a dark->light edge
K_right = -K_left # the mirror image
def conv(a, k):
p = np.pad(a, ((1, 1), (1, 1)), mode="edge")
return np.array([[float((p[i:i + 3, j:j + 3] * k).sum())
for j in range(a.shape[1])] for i in range(a.shape[0])])
fm = conv(img, K_left)
print("A FEATURE MAP from one edge kernel, row 3:")
print(" " + " ".join("%6.1f" % v for v in fm[3]))
print(" positive at the LEFT edge of the bar (dark to light) and negative")
print(" at the right edge (light to dark). same kernel, opposite signs.")
print()
print("AFTER ReLU -- max(x, 0):")
print(" " + " ".join("%6.1f" % v for v in np.maximum(fm[3], 0)))
print(" the right edge is gone. not attenuated -- gone, exactly zero.")
print(" this feature now means 'dark-to-light edge here', a claim that")
print(" is either made or not made. before ReLU it meant 'dark-to-light")
print(" by this much, OR light-to-dark by this much', which is two")
print(" different things sharing one number.")
print()
print(" the network's answer to losing the other edge is not to recover")
print(" it -- it is to learn a SECOND kernel that is the mirror image:")
fm2 = conv(img, K_right)
print(" mirrored kernel + ReLU, row 3:")
print(" " + " ".join("%6.1f" % v for v in np.maximum(fm2[3], 0)))
print(" this is why filter counts are what they are. a linear layer needs")
print(" one filter per direction of variation; a ReLU layer needs one per")
print(" DIRECTED half of it, so a ReLU network needs roughly twice the")
print(" channels to represent the same variation -- a real cost, paid")
print(" because the resulting features are far easier to build on.")
print()
print("SPARSITY -- how much of a feature map survives:")
print("%-30s %12s %12s" % ("", "nonzero", "fraction"))
for label, a in (("before ReLU", fm), ("after ReLU", np.maximum(fm, 0))):
nz = int((np.abs(a) > 1e-9).sum())
print("%-30s %12d %11.1f%%" % (label, nz, 100.0 * nz / a.size))
print()
print("NOW THE POINT THAT MATTERS. WITHOUT a nonlinearity, stacking layers")
print("buys you nothing at all. two convolutions in a row, no activation:")
A = rng.normal(0, 1, (3, 3))
B = rng.normal(0, 1, (3, 3))
x = rng.normal(0, 1, (9, 9))
two_layers = conv(conv(x, A), B)
# the composition of two convolutions is a single convolution with the
# kernels convolved together -- build that 5x5 kernel explicitly
C = np.zeros((5, 5))
for i in range(3):
for j in range(3):
C[i:i + 3, j:j + 3] += B[i, j] * A
def conv5(a, k):
p = np.pad(a, ((2, 2), (2, 2)), mode="edge")
return np.array([[float((p[i:i + 5, j:j + 5] * k).sum())
for j in range(a.shape[1])] for i in range(a.shape[0])])
one_layer = conv5(x, C)
inner = slice(3, 6)
print(" two 3x3 convolutions, interior values:")
print(" " + " ".join("%8.4f" % v for v in two_layers[4][inner]))
print(" ONE 5x5 convolution with the combined kernel:")
print(" " + " ".join("%8.4f" % v for v in one_layer[4][inner]))
print(" max difference in the interior: %.2e"
% np.abs(two_layers[2:-2, 2:-2] - one_layer[2:-2, 2:-2]).max())
print(" identical. a hundred stacked linear layers collapse into ONE")
print(" linear layer, exactly, always. depth without a nonlinearity is")
print(" not a deep model -- it is an expensive way to write a shallow one.")
print()
print(" now put a ReLU between them and try to collapse it:")
relu_stack = conv(np.maximum(conv(x, A), 0), B)
print(" with ReLU: " + " ".join("%8.4f" % v for v in relu_stack[4][inner]))
print(" difference from the single 5x5: %.4f"
% np.abs(relu_stack[2:-2, 2:-2] - one_layer[2:-2, 2:-2]).max())
print(" no 5x5 kernel reproduces it, because the function is no longer")
print(" linear. THAT is what the activation buys: it makes depth mean")
print(" something.")
print()
print("WHY ReLU AND NOT A SMOOTH CURVE. compare gradients:")
print("%-16s %14s %14s %14s" % ("input x", "sigmoid'(x)", "tanh'(x)", "ReLU'(x)"))
for v in (-6.0, -2.0, 0.0, 2.0, 6.0):
sg = 1 / (1 + np.exp(-v))
print("%-16.1f %14.6f %14.6f %14.0f"
% (v, sg * (1 - sg), 1 - np.tanh(v) ** 2, 1.0 if v > 0 else 0.0))
print(" ReLU is not differentiable at exactly 0; every framework simply")
print(" picks 0 there, and it has never mattered in practice.")
print(" sigmoid's gradient at x=6 is %.6f. multiply ten of those together"
% (1 / (1 + np.exp(-6.0)) * (1 - 1 / (1 + np.exp(-6.0)))))
print(" for a ten-layer network and the gradient reaching layer 1 is")
print(" %.2e -- the vanishing gradient, and it is why nothing deep"
% (1 / (1 + np.exp(-6.0)) * (1 - 1 / (1 + np.exp(-6.0)))) ** 10)
print(" trained before ReLU. ReLU's gradient is exactly 1 wherever the")
print(" unit is active, so it passes gradient through unchanged, however")
print(" many layers deep.")
print()
print("AND THE FAILURE IT INTRODUCES -- DEAD UNITS. ReLU's gradient is")
print("exactly 0 on the other side. a unit whose bias drifts far negative")
print("outputs 0 for every input, so it gets 0 gradient, so it can never")
print("recover:")
z = rng.normal(0, 1, 2000)
print("%-24s %14s %14s" % ("bias", "fraction active", "gradient flow"))
for bias in (0.5, 0.0, -1.0, -3.0, -6.0):
act = (z + bias) > 0
print("%-24.1f %14.3f %14s"
% (bias, act.mean(), "dead" if act.mean() == 0 else "ok"))
print(" at bias -6 not one of %d inputs activates the unit. it is a" % z.size)
print(" constant 0 that will stay a constant 0 forever, and it still")
print(" costs memory and multiplications on every forward pass.")
print(" the fixes all keep a little slope on the left: LeakyReLU uses")
print(" 0.01x, ELU and GELU curve smoothly through the origin. all of")
print(" them trade a little of ReLU's exact sparsity for the guarantee")
print(" that no unit is ever permanently unreachable.")
Output
Worth remembering
ReLU stands for Rectified Linear Unit.
It replaces negative values with 0 and keeps positive values unchanged.
It provides essential non-linearity to CNNs.
It is computationally extremely cheap compared to functions like Sigmoid or Tanh.
It promotes network sparsity (many inactive nodes).
The variants, and when they help
Activation
Negative side
Notes
ReLU
Exactly 0
The default; fast; can die
Leaky ReLU
0.01x
Keeps a small gradient alive
PReLU
αx, with α learned
Slightly better, slightly more to fit
ELU
α(eˣ − 1)
Smooth, mean closer to zero, more expensive
GELU
x · Φ(x)
Smooth; standard in transformers
SiLU / Swish
x · sigmoid(x)
Smooth; common in modern vision models
For convolutional networks, plain ReLU remains the sensible default. Leaky ReLU is the first thing to try if you observe many dead units. GELU and SiLU dominate in transformers and in recent efficient vision architectures, where their smoothness appears to help slightly — at a small computational cost.
Preventing dead units matters more than choosing among the variants:
Use He initialisation, which is designed for ReLU and scales the initial weights by √(2/fan_in). Xavier initialisation, designed for tanh, leaves ReLU networks with activations that shrink through depth.
Lower the learning rate if you see a sudden collapse in the fraction of active units.
Add batch normalisation before the activation, which keeps pre-activations centred and makes the dead region much harder to fall into.
Where ReLU sits in a block
The conventional order in a convolutional network is:
Conv → BatchNorm → ReLU
Batch normalisation before the activation is the arrangement the original paper used and it remains standard. It keeps the input to ReLU centred so that roughly half the units are active, which is what the initialisation assumed.
Note that the convolution's bias is redundant when batch normalisation follows, because the normalisation subtracts the mean anyway — which is why bias=False is common on convolutions in ResNet-style code.
In residual blocks the placement is more delicate: the pre-activation variant (BN → ReLU → Conv) trains deeper networks better, because it leaves the identity path completely clean, and that detail was worth a paper of its own.
Questions people ask
How do I know if my ReLUs are dying? Log the fraction of zero activations per layer. Around 50% is healthy; 90%+ that keeps rising is a problem.
Is ReLU differentiable at zero? Not strictly — there is a corner. Frameworks define the derivative there as 0 by convention, and it causes no practical trouble because exactly zero almost never occurs.
Should I use ReLU on the output layer? Only for regression targets that must be non-negative. Classification uses softmax or sigmoid; unbounded regression uses no activation.
Does ReLU cause exploding gradients? It does not damp them, since the positive-side derivative is 1. Combined with poor initialisation or a high learning rate that can allow explosion, which is what gradient clipping and normalisation address.
Why does ReLU work despite discarding half the information? Because the network compensates by learning filters in both polarities, and the sparsity it produces appears to be useful rather than merely lossy.
Is GELU better than ReLU? Marginally, in transformers. In convolutional networks the difference is usually within noise, and ReLU is faster.
Recap in one screen
max(0, x): negatives become zero, positives pass through.
Its derivative is exactly 1 on the positive side, which is what lets gradients survive depth.
Without a non-linearity, stacked layers collapse into one linear map.
It is fast and produces sparse activations; the cost is dead neurons on the negative side.
He initialisation, batch normalisation and a sensible learning rate prevent most dying-ReLU problems.
Conv → BatchNorm → ReLU is the standard block order.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
For convolutional networks, plain ReLU remains the sensible default. Leaky ReLU is the first thing to try if you observe many dead units. GELU and SiLU dominate in transformers and in recent efficient vision architectures, where their smoothness appears to help slightly — at a small computational cost.
What does this module say about “What is an Activation Function”?
When a Convolutional or Dense layer processes data, it computes a series of linear mathematical operations (dot products). If a neural network only consisted of linear operations, stacking multiple layers wouldn't help—the entire network would mathematically collapse into a single linear model. To learn complex, real-world patterns (like recognizing a face or a dog), we must introduce non-linearity .
What does this module say about “Enter ReLU (Rectified Linear Unit)”?
ReLU is currently the most popular activation function in the world for hidden layers. Its formula is brilliantly simple: if the input is negative, output 0. If the input is positive, output the input unchanged.
Cheat sheet
ReLU Activation in CNN
Visualize the Rectified Linear Unit (ReLU). See how it introduces non-linearity by passing positive signals and turning off (clipping) negative signals.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.