Activation Explorer

The four you will actually meet

• Sigmoid — 1 / (1 + e^−x), output in (0, 1). Reads as a probability, which is why it survives at the output of binary classifiers.
• Tanh — output in (−1, 1) and centred on zero, which tends to train better than sigmoid in hidden layers.
• ReLU — max(0, x). Gradient is exactly 1 for positive input and exactly 0 otherwise. Cheap, and the default for hidden layers.
• Leaky ReLU — max(0.01x, x). Keeps a small gradient alive on the negative side.

Numbers at three inputs

Definitions blur together; a table of actual outputs does not. For x = −2, 0 and 3:

Three things fall out of that table. Sigmoid and tanh both flatten at the extremes, so their derivatives there are near zero. ReLU discards the negative side entirely and passes the positive side through untouched. And GELU behaves almost identically to ReLU for large inputs while staying smooth near zero.

The derivatives are what decide trainability:

Sigmoid's ceiling of 0.25 is the reason deep sigmoid networks do not train: ten layers multiply to 0.25¹⁰ ≈ one in a million, and the early layers stop receiving a usable signal.

The dying ReLU problem

ReLU outputs zero for any negative input, and its derivative there is also zero. If a neuron’s weights are pushed far enough negative that it outputs zero for every training example, it receives no gradient — and with no gradient it can never recover. The neuron is permanently dead.

This is not rare. A large learning rate can kill a substantial fraction of a layer in a single update, and dead neurons are invisible in the loss curve; the network simply has less capacity than you think it does.

Which one to use where

• Hidden layers, general default — ReLU. Cheap, no vanishing gradient in the positive region, and works.
• Hidden layers, if neurons are dying — Leaky ReLU or ELU.
• Transformers — GELU, essentially universally.
• Recurrent networks — tanh inside the cell, because bounded outputs keep the recurrent state from growing without limit.
• Anywhere deep, with sigmoid — avoid. Its maximum derivative of 0.25 shrinks the gradient by at least a factor of four per layer.

Output activations are a separate decision

Hidden-layer activations exist to introduce non-linearity. Output activations exist to put the prediction in the right range, and the choice is dictated by the task rather than by preference:

• Regression — no activation. The output must be unbounded.
• Binary classification — sigmoid, giving one probability in [0, 1].
• Multi-class, one label — softmax, giving probabilities across classes that sum to 1.
• Multi-label — sigmoid on each output independently, since several labels can be true at once and the outputs should not sum to 1.

One practical warning: frameworks usually fold the output activation into the loss for numerical stability, so CrossEntropyLoss in PyTorch expects raw logits and applies softmax itself. Adding your own softmax before it applies the function twice and quietly degrades training.

Choosing one

The multi-label row is worth stating plainly: softmax forces the outputs to sum to 1, which is wrong when several labels can be true at once. Using it there is a modelling error, not a preference.

And "output: none" is deliberate. Frameworks apply softmax or sigmoid inside the loss function for numerical stability, so adding it in the model as well applies it twice — a real bug that trains slowly and silently.

The dying ReLU problem, and its fixes

A ReLU unit whose pre-activation is always negative outputs zero, has a derivative of zero, receives no gradient, and never recovers. It is dead for the rest of training.

A network can lose a substantial fraction of its units this way, usually after a learning rate large enough to push weights into a bad region.

The responses, in order of usefulness:

• He initialisation — scale initial weights by √(2/fan_in). It is designed for ReLU; Xavier initialisation, designed for tanh, leaves activations shrinking through depth.
• A lower learning rate. A sudden collapse in the fraction of active units is the signature.
• Batch normalisation before the activation, which keeps pre-activations centred so about half the units are active.
• Leaky ReLU (0.01x on the negative side), which keeps a small gradient alive.

Diagnose it by logging the fraction of zero activations per layer. Around 50% is healthy; 90% and rising is a problem.

Try this above

1. Drag Input Value (x) to around 6 and look at sigmoid and tanh. Both are pinned near their ceiling and the curve is nearly flat — that flatness is saturation.
2. Bring x back toward 0. Both functions are at their steepest here, which is where they learn fastest.
3. Set x to −3 and note ReLU outputs exactly 0, not a small number.

What usually goes wrong

Sigmoid in deep hidden layers. Its derivative peaks at 0.25, so gradients shrink by at least 4× per layer. Five layers of sigmoid multiplies the gradient by roughly 0.25⁵ ≈ 0.001 in the best case — the early layers barely move. This is the vanishing gradient problem, and it is why ReLU became standard.Dying ReLU. A neuron pushed firmly negative outputs 0 and has zero gradient, so nothing ever pulls it back. It is dead for the rest of training. Leaky ReLU exists precisely to leave a small escape route.

In one line

ReLU in the hidden layers by default; sigmoid or softmax only at the output, where the bounded range is the point.

The smooth modern variants

GELU multiplies the input by the probability that a standard normal variable is below it — effectively a soft, probabilistic gate. It is smooth everywhere, allows small negative values through, and is the standard in BERT, GPT and most transformers.

SiLU (Swish) is x · sigmoid(x) — very similar in shape, slightly cheaper, and common in EfficientNet and modern YOLO variants.

Mish, ELU, SELU each have theoretical arguments and appear occasionally. None has displaced the three above.

Is the difference worth caring about? For convolutional networks, usually not — ReLU is faster and within noise of the alternatives. For transformers, GELU is the convention and the small gain appears real. The bigger wins are elsewhere: architecture, data, learning rate, normalisation.

Where they sit in a block

The conventional order in a convolutional network is Conv → BatchNorm → ReLU, with normalisation before the activation so the input to ReLU stays centred.

In transformers the arrangement differs: LayerNorm → Attention → residual, then LayerNorm → FFN(GELU) → residual. The pre-normalisation variant, with normalisation before each sublayer rather than after, is what makes very deep transformers trainable.

A detail worth knowing: when batch normalisation follows a convolution, the convolution's bias is redundant, because normalisation subtracts the mean anyway. That is why bias=False appears on convolutions in ResNet-style code.

Six activations, and the one property they exist for

Without a non-linear activation a hundred layers collapse into one matrix. This proves that, then compares what each function does to a gradient.

import numpy as np

rng = np.random.default_rng(0)

print("first, why they are needed at all. stack three LINEAR layers:")
A = rng.normal(0, 0.5, (4, 6))
B = rng.normal(0, 0.5, (6, 5))
C = rng.normal(0, 0.5, (5, 3))
x = rng.normal(size=(2, 4))
three = ((x @ A) @ B) @ C
one = x @ (A @ B @ C)
print("  x @ A @ B @ C  =", np.round(three[0], 6))
print("  x @ (A @ B @ C)=", np.round(one[0], 6))
print("  identical:", np.allclose(three, one))
print("  three layers, %d parameters, and it is EXACTLY equivalent to one"
      % (A.size + B.size + C.size))
print("  4x3 matrix with %d. depth bought nothing." % (4 * 3))
print()

acts = {
    "sigmoid":    (lambda z: 1 / (1 + np.exp(-z)),
                   lambda z: (1 / (1 + np.exp(-z))) * (1 - 1 / (1 + np.exp(-z)))),
    "tanh":       (np.tanh, lambda z: 1 - np.tanh(z) ** 2),
    "relu":       (lambda z: np.maximum(0, z), lambda z: (z > 0).astype(float)),
    "leaky relu": (lambda z: np.where(z > 0, z, 0.01 * z),
                   lambda z: np.where(z > 0, 1.0, 0.01)),
    "gelu":       (lambda z: 0.5 * z * (1 + np.tanh(np.sqrt(2 / np.pi) *
                                                    (z + 0.044715 * z ** 3))),
                   None),
    "softplus":   (lambda z: np.log1p(np.exp(-np.abs(z))) + np.maximum(z, 0),
                   lambda z: 1 / (1 + np.exp(-z))),
}

zs = np.array([-4.0, -1.0, -0.1, 0.0, 0.1, 1.0, 4.0])
print("the functions themselves:")
print("%12s %s" % ("z", "".join("%9.1f" % z for z in zs)))
for name, (f, _) in acts.items():
    print("%12s %s" % (name, "".join("%9.4f" % v for v in f(zs))))
print()

print("and their gradients -- this is the column that decides training:")
print("%12s %s" % ("z", "".join("%9.1f" % z for z in zs)))
h = 1e-6
for name, (f, d) in acts.items():
    g = d(zs) if d else (f(zs + h) - f(zs - h)) / (2 * h)
    print("%12s %s" % (name, "".join("%9.4f" % v for v in g)))
print()
print("look at the softplus gradient row and the sigmoid FUNCTION row above --")
print("they are identical. the derivative of softplus is exactly the sigmoid,")
print("which is a pleasant thing to discover rather than be told.")
print()
print("read the row for sigmoid. at z=4 the gradient is %.4f. multiply ten of"
      % ((1 / (1 + np.exp(-4))) * (1 - 1 / (1 + np.exp(-4)))))
print("those together for a ten-layer network and you get %.2e -- the signal"
      % ((1 / (1 + np.exp(-4)) * (1 - 1 / (1 + np.exp(-4)))) ** 10))
print("reaching the first layer is gone. that is the vanishing gradient, and")
print("it is why sigmoid stopped being used in hidden layers.")
print()
print("relu's gradient is exactly 1 wherever it is positive, so stacking does")
print("not shrink it. the price is the other half:")
z = rng.normal(-0.5, 1.0, 10000)
print("  with inputs centred at -0.5, %.1f%% of relu units output exactly 0"
      % (100 * (z <= 0).mean()))
print("  and a unit stuck at 0 has gradient 0, so it never updates again.")
print("  that is a 'dead relu'. leaky relu's 0.01 slope is the cheap fix.")
print()

print("max magnitude of each gradient -- how much signal survives one layer:")
for name, (f, d) in acts.items():
    g = d(np.linspace(-6, 6, 2001)) if d else None
    if g is None:
        zz = np.linspace(-6, 6, 2001)
        g = (f(zz + h) - f(zz - h)) / (2 * h)
    print("  %-12s %.4f" % (name, g.max()))
print()
print("the practical summary: relu or gelu in hidden layers. sigmoid only on")
print("a binary output, softmax only on a multi-class output. tanh survives in")
print("recurrent nets, where its zero-centred output genuinely helps.")

Questions people ask

Why not use a linear activation? Because stacked linear layers collapse into one matrix, so the network could only ever draw a straight boundary.

Is ReLU differentiable at zero? Strictly no — there is a corner. Frameworks define it as 0 by convention, and it causes no practical trouble.

Should I use sigmoid in hidden layers? No. Its derivative ceiling of 0.25 causes vanishing gradients. It survives only at outputs.

Is tanh better than sigmoid? In hidden layers, yes — it is zero-centred and its derivative reaches 1. Both are still worse than ReLU for depth.

Do different layers need different activations? Rarely. One for the hidden layers and one for the output covers almost every architecture.

What about learnable activations? PReLU learns the negative slope, and it helps marginally at the cost of extra parameters. Not usually worth the complexity.

Recap in one screen

• Activations are what stop stacked layers collapsing into a single linear map.
• Sigmoid's derivative peaks at 0.25, which is why deep sigmoid networks do not train.
• ReLU passes a derivative of exactly 1 on the positive side — that is what makes depth work.
• Its cost is dead units; He initialisation, normalisation and a sensible learning rate prevent most of them.
• GELU and SiLU are the smooth modern choices, standard in transformers and recent vision models.
• Leave the output layer unactivated and let the loss function apply softmax or sigmoid.