The Perceptron

Work one through by hand

Take two inputs with weights w = [0.6, −0.4] and bias b = 0.1. Feed it x = [1.0, 0.5]:

z = (0.6 × 1.0) + (−0.4 × 0.5) + 0.1 = 0.6 − 0.2 + 0.1 = 0.5

With a Step activation the output is 1, because z is above zero. With Sigmoid the same z gives 1 / (1 + e^−0.5) = 0.62 — the same decision, but now with a sense of how confident it is. That difference is the whole reason step activations were abandoned.

Two inputs worked through by hand

Take weights w = [0.6, −0.4], bias b = 0.1, and input x = [1.0, 0.5]:

z = (0.6 × 1.0) + (−0.4 × 0.5) + 0.1 = 0.6 − 0.2 + 0.1 = 0.5

With a step activation the output is 1, because z is above zero. With sigmoid, the same z gives 1/(1 + e⁻⁰‧⁵) = 0.62 — the same decision, now with a sense of how confident it is.

That difference is the whole reason step activations were abandoned. A step function's gradient is zero everywhere it is defined, so gradient descent has nothing to follow, and the network cannot be trained by backpropagation at all.

Change one number and watch the decision flip. With b = −0.7 instead: z = 0.6 − 0.2 − 0.7 = −0.3, and the output becomes 0. Same inputs, same weights, opposite answer — which is exactly what the bias is for.

What the two parts do geometrically

• Weights rotate. They set which direction in input space the unit is sensitive to, and how strongly. A negative weight inverts the unit's opinion of that feature.
• The bias translates. It slides the decision boundary without turning it. Without a bias, every boundary is forced through the origin — a severe and usually pointless restriction.

For two inputs the boundary is the line where z = 0, which is w₁x₁ + w₂x₂ + b = 0. That is the equation of a straight line, with the weights giving its orientation and the bias its offset. In three dimensions it is a plane; beyond that a hyperplane.

So a perceptron is a linear classifier, and everything it can and cannot do follows from that.

The XOR limitation, and why it mattered historically

One perceptron draws exactly one straight boundary. That is enough for AND and OR:

Plot XOR's four points and no single straight line separates the ones from the zeros. The two positive cases sit on opposite corners. No assignment of weights and bias will ever solve it — this is a proof, not a training difficulty.

Minsky and Papert made this argument in 1969, and it contributed to a long slowdown in neural network research. The resolution was already implicit: stack perceptrons into layers. Two units in a hidden layer can carve the space into regions that a third unit can then combine, and XOR becomes solvable.

What was genuinely missing until the 1980s was a way to train those layers — which required a differentiable activation and the chain rule, that is, sigmoid and backpropagation.

Try this above

1. Set Number of Inputs to 2 and pick Step (Binary). The output can only ever be 0 or 1.
2. Switch to Sigmoid (Smooth) without changing anything else. Same weighted sum, but the output now slides continuously.
3. Switch to ReLU (Rectified) and make z negative. The output pins to exactly 0 and stays there.
4. Raise Number of Inputs to 5 and count the weights — one per input, plus a single bias regardless.

Where a single perceptron fails

One perceptron draws exactly one straight boundary. That is enough for AND and OR, and famously not enough for XOR — no single line separates XOR's two classes, so no assignment of weights and bias will ever solve it. This limitation stalled neural network research for years; the answer was to stack perceptrons into layers.

The other trap is the Step activation itself. Its gradient is zero everywhere it is defined, so gradient descent has nothing to follow. You cannot train a step-activated network by backpropagation, which is why sigmoid and later ReLU took over.

In one line

One weighted sum, one bias, one activation, one straight boundary — and a hard ceiling on what that can express.

From perceptron to modern neuron

The modern unit is the same shape with two substitutions.

The original perceptron learning rule was simple and only worked for separable data: when a prediction is wrong, add the input to the weights (or subtract it), and repeat. On non-separable data it never settles.

Replacing the step with a smooth function is what unlocked everything else, because it gave the weights a gradient to follow. That single change is the difference between a historical curiosity and the building block of every model in use today.

Worth noting what did not change: one weighted sum, one bias, one non-linearity. A unit in a 400-billion-parameter model computes exactly what a 1958 perceptron computed, with a different activation.

Counting the parameters

A perceptron with n inputs has n weights and 1 bias — n + 1 parameters, regardless of how many examples it sees.

Extend to a layer of m units and it is m × (n + 1). For 100 inputs and 50 units: 50 × 101 = 5,050 parameters. That formula, applied layer by layer, is how every network's size is counted.

The bias is one per unit, not one per weight — a detail people get wrong when counting by hand.

Train one, and find the thing it cannot learn

A perceptron is a weighted sum and a threshold. Twelve lines train it, and four rows of data defeat it -- which is the whole reason the rest of this track exists.

import numpy as np

def train(X, y, epochs=30, lr=0.1):
    w = np.zeros(X.shape[1])
    b = 0.0
    for epoch in range(epochs):
        errors = 0
        for xi, target in zip(X, y):
            pred = 1 if xi @ w + b > 0 else 0
            update = lr * (target - pred)
            w += update * xi
            b += update
            errors += int(update != 0.0)
        if errors == 0:
            return w, b, epoch + 1
    return w, b, epochs

print("AND -- linearly separable:")
X = np.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y_and = np.array([0, 0, 0, 1])
w, b, ep = train(X, y_and)
print("  converged in %d epochs. w = %s, b = %.2f" % (ep, np.round(w, 2), b))
for xi, t in zip(X, y_and):
    z = xi @ w + b
    print("    %s -> z %+6.2f -> %d   (wanted %d)" % (xi, z, int(z > 0), t))
print()

print("OR -- also separable:")
y_or = np.array([0, 1, 1, 1])
w, b, ep = train(X, y_or)
print("  converged in %d epochs. w = %s, b = %.2f" % (ep, np.round(w, 2), b))
print()

print("XOR -- and here is the wall:")
y_xor = np.array([0, 1, 1, 0])
w, b, ep = train(X, y_xor, epochs=5000)
print("  after %d epochs it has not converged. w = %s, b = %.2f"
      % (ep, np.round(w, 2), b))
correct = sum(int((xi @ w + b > 0)) == t for xi, t in zip(X, y_xor))
print("  it gets %d of 4 right, which is what guessing gets you." % correct)
print()
print("this is not a training problem, it is an arithmetic one. a perceptron")
print("draws ONE straight line. XOR needs the two diagonal corners on one side")
print("and the other two on the other, and no straight line does that:")
for xi, t in zip(X, y_xor):
    print("    %s wants %d" % (xi, t))
print()

print("the fix is a hidden layer -- two lines instead of one:")
W1 = np.array([[1.0, 1.0], [1.0, 1.0]])
b1 = np.array([-0.5, -1.5])        # neuron 1 fires for OR, neuron 2 for AND
W2 = np.array([1.0, -2.0])
b2 = -0.5
for xi, t in zip(X, y_xor):
    h = (xi @ W1 + b1 > 0).astype(float)     # step activation
    out = int(h @ W2 + b2 > 0)
    print("    %s -> hidden %s -> %d   (wanted %d)" % (xi, h, out, t))
print()
print("neuron 1 computes OR, neuron 2 computes AND, and the output layer says")
print("'OR but not AND' -- which is exactly XOR. one layer of composition")
print("turned an impossible problem into a trivial one.")
print()
print("the historical note that matters: this limitation was published in 1969")
print("and stalled the field for a decade. the hidden layer was not the hard")
print("part -- training one was, and that needed backpropagation.")

Questions people ask

Is a perceptron the same as a neuron? Historically the perceptron used a step activation. "Neuron" or "unit" today means the same structure with a smooth activation.

Why can it not solve XOR? Because XOR is not linearly separable, and one perceptron can only draw one straight boundary.

How does a multi-layer network solve it? The hidden layer transforms the input space into one where the classes are linearly separable, and the output unit then draws the line.

Can a perceptron do regression? With no activation, z = wx + b is exactly linear regression. Add a sigmoid and it is logistic regression.

Is the bias necessary? Almost always. Without it, the boundary must pass through the origin, which rules out most useful solutions.

What does a negative weight mean? That the feature counts against the unit firing. Weights carry direction as well as strength.

Recap in one screen

• One weighted sum, one bias, one activation — the smallest complete unit of a network.
• Weights set the boundary's orientation; the bias slides it away from the origin.
• The output is a straight-line decision, so AND and OR are solvable and XOR is not.
• A step activation has zero gradient everywhere, which is why it cannot be trained by backpropagation.
• Modern units are identical in structure with a smooth activation — that substitution is what made depth trainable.