Visualize the Rectified Linear Unit (ReLU). See how it introduces non-linearity by passing positive signals and turning off (clipping) negative signals.
ReLU Function Graph
PROCESSING...
Mathematical Formula
A = max(0, Z)
Click "Forward Pass" to view calculations.
Understanding the ReLU Activation Function
The simple mathematical trick that powers modern Deep Learning.
What is an Activation Function?
When a Convolutional or Dense layer processes data, it computes a series of linear mathematical operations (dot products). If a neural network only consisted of linear operations, stacking multiple layers wouldn't help—the entire network would mathematically collapse into a single linear model. To learn complex, real-world patterns (like recognizing a face or a dog), we must introduce non-linearity. This is the job of the Activation Function.
Enter ReLU (Rectified Linear Unit)
ReLU is currently the most popular activation function in the world for hidden layers. Its formula is brilliantly simple: if the input is negative, output 0. If the input is positive, output the input unchanged.
Mathematically: f(x) = max(0, x)
In a biological sense, think of a node as a neuron. If the incoming signal (pre-activation) is negative, the neuron decides it's not relevant and does not "fire" (outputs 0). If the signal is positive, it fires proportionately to the signal's strength.
Guided Experiments with This Interactive
Observe the Clipping:
Look at the Pre-Activation sliders on the left. Set a few to negative numbers and a few to positive. Notice the Outputs on the right immediately clip the negative ones to exactly 0.00 (turning grey), while the positive ones carry over perfectly.
Watch the Forward Pass:
Click the Forward Pass button. Watch the animation on the center canvas. You will see exactly where each input lands on the $y = \max(0, x)$ curve. Notice that any point on the left side of the Y-axis drops straight to the bottom floor line.
Identify Sparsity:
Click the "Randomize Values" button a few times. On average, about half of the outputs will be zero. This creates a "sparse" network where only a subset of neurons is active at any given time. This makes the network highly efficient and helps prevent neurons from becoming overly dependent on each other.
The "Dying ReLU" Problem
Because ReLU outputs exactly 0 for any negative input, its gradient (slope) for negative values is also 0. During backpropagation, if a neuron consistently receives negative inputs, its weights will never update. It becomes a "dead" neuron.
To solve this, variants like Leaky ReLU were invented, which allow a tiny, non-zero gradient for negative numbers (e.g., $f(x) = 0.01x$ when $x < 0$).
Key Takeaways
ReLU stands for Rectified Linear Unit.
It replaces negative values with 0 and keeps positive values unchanged.
It provides essential non-linearity to CNNs.
It is computationally extremely cheap compared to functions like Sigmoid or Tanh.
It promotes network sparsity (many inactive nodes).
Cheat sheet
ReLU Activation in CNN
Visualize the Rectified Linear Unit (ReLU). See how it introduces non-linearity by passing positive signals and turning off (clipping) negative signals.