Weights & Biases

What each one does geometrically

A neuron computes z = w · x + b. Those two terms do other jobs, and it is worth separating them:

• Weights rotate. They set the orientation of the decision edge — which way in input space the neuron is sensitive to. And how strongly.
• The bias translates. It slides the edge without turning it. Without a bias each edge is forced through the origin. It is a severe and often pointless restriction.

Together they set a linear function. In a single neuron with n inputs, the weights are n counts and the bias is one count, giving n + 1 learnable params. A fully connected layer with 784 inputs and 256 outputs has 784 × 256 weights plus 256 biases, totalling 200,960 params. A deep network stacks many such layers. And the total param count — tens of millions for a modest net, billions for a large one — is almost fully weights and biases.

Training adjusts these counts. The forward pass computes predictions with the current weights and biases. The loss function measures how wrong the predictions are. Backpropagation computes the gradient of the loss with respect to each weight and each bias. The optimiser (SGD, Adam) nudges each param in the way that reduces the loss. This cycle repeats for each batch of training data. And over thousands of iterations the weights and biases converge toward vals that make the network's output match the training labels.

Initialisation matters since the starting vals of weights find whether gradients flow or vanish. Setting all weights to zero makes each neuron identical. So the network cannot learn other features — this is the symmetry-breaking problem. The standard fix is random initialisation scaled to the layer width: Xavier initialisation for tanh/sigmoid activations, He initialisation for ReLU. Biases are often initialised to zero. It is fine since the weights now break symmetry.

A concrete case

One input, weight w = 2.0, bias b = 0. At x = 0.5 you get z = 1.0 — the neuron fires.

Now set b = −1.5 and leave all else alone: z = 1.0 − 1.5 = −0.5. Same input, same weight, opposite decision. The bias moved the threshold from x > 0 to x > 0.75.

This is why a bias is not optional detail. It is the gap between "is this input positive?" and "is this input above the level that really matters?"

Why the bias cannot be dropped

Without a bias, the weighted sum is zero whenever each input is zero. So the decision edge is forced through the origin. That is a severe restriction: a neuron that should fire just when its input exceeds 5 cannot express that threshold at all, since it has no way to shift its output independently of the inputs.

The bias supplies just that freedom. In y = wx + b, w tilts the line and b slides it — and the same is true in a thousand dimensions. There the weights orient a hyperplane and the bias offsets it from the origin. One extra param per neuron buys the ability to place the edge anywhere rather than just through a single fixed point.

Counting a layer's params

A fully connected layer with n_in inputs and n_out outputs has one weight per connection plus one bias per output neuron:

params = (n_in × n_out) + n_out

For 784 inputs and 128 outputs that is 784 × 128 + 128 = 100,480. The biases are 128 of them — about 0.1% of the total. It is why they are cheap enough to always include and why regularisation often skips them.

Notice the weight count is a product. Doubling either the input or the output size doubles the params. Doubling both quadruples them. That quadratic growth is why wide layers dominate a network’s memory footprint.

What training really changes

Training changes nothing about the network except these counts. The design, the activations and the connections are all fixed before the first step. Gradient descent just ever adjusts weights and biases.

That is worth holding onto, since it clarifies what a trained net is: a set set of vals for these params. Saving a net saves the weights and biases. Transfer learning reuses another net’s weights and biases. A randomly initialised network and a fully trained one differ in no other respect.

One concrete case

One input, weight w = 2.0, bias b = 0. At x = 0.5 you get z = 1.0 — the neuron fires.

Now set b = −1.5 and change nothing else: z = 1.0 − 1.5 = −0.5. Same input, same weight, opposite decision.

The bias moved the threshold. Before, the neuron fired whenever x > 0. Now it fires whenever x > 0.75. That is the gap between asking "is this input positive?" and "is this input above the level that really matters?"

Which is why a bias is not an optional detail. A network without biases can just draw edges through the origin. And almost no real decision edge passes through the origin.

What they are, and how many there are

Weights are the multipliers on the connections between units. A layer taking 100 inputs into 50 units has a 100×50 weight matrix — 5,000 counts, one for each input-output pair.

Biases are one count per unit — 50 in that layer — added after the weighted sum.

z = w · x + b

Both are learned. All else you set — the count of layers, the learning rate, the batch size — is in service of finding good vals for these two.

A useful proportion: biases are a tiny fraction of a network's params. A layer with 100 inputs and 50 units has 5,000 weights and 50 biases. So about 1%. They are cheap and they matter.

Reading trained weights, and the limits of doing so

In a single-layer net, a weight's magnitude is interpretable: a large positive coefficient on "smoking" means smoking pushes the prediction up. That reading is why logistic regression survives in regulated settings.

In anything deeper, it breaks down. A feature's influence is spread across many paths. And a small first-layer weight can still matter enormously downstream through amplification in later layers. Reading a single weight as feature weight in a deep network is unreliable. And integrated gradients or SHAP are the tools that do the job well.

Two things that are worth reading from trained weights:

Wildly other magnitudes across features is almost always a symptom of unscaled inputs, not a finding. If one feature is measured in thousands and another in decimals, the network compensates with tiny and huge weights. And gradient descent struggles to move both usefully. Scale the inputs instead.

The distribution of weight magnitudes tells you whether regularisation is doing anything. A histogram concentrated near zero with a few large vals is what weight decay gives.

Try this above

1. Set Count of Inputs to 1 so there is a single weight to watch.
2. Change that weight and watch the response curve steepen and flip way. Negative weights invert the neuron's opinion of that feature.
3. Now change just the bias. The shape stays identical; it slides sideways.
4. Switch Activation Function between Step and Sigmoid and repeat — the geometry is the same, just the sharpness changes.

What often goes wrong

Wildly other weight magnitudes are almost always a symptom of unscaled inputs rather than a real finding. If one feature is measured in thousands and another in decimals, the network compensates with tiny and huge weights. And gradient descent struggles to move both usefully. Scale the inputs instead.Reading a single weight as feature weight is unreliable in anything deeper than one layer. In a multi-layer network a feature's influence is spread across many paths. And a small first-layer weight can still matter enormously downstream.

In one line

Weights decide way and strength. The bias decides where the threshold sits.

Where the bias is left out on purpose

There are two standard cases where a bias is omitted.

Before batch normalisation. Normalisation subtracts the mean, which cancels any constant the last layer added. So a convolution followed by batch normalisation has a redundant bias. And reference implementations write bias=False. It saves params and changes nothing.

In some attention projections, where recent language nets drop biases from the query/key/value projections and sometimes from the feed-forward layers as well — they turn out to add little at scale. And removing them simplifies the net slightly.

Otherwise, keep the bias. The default in each framework includes it for good reason.

Their role in training

At initialisation, weights are small random vals and biases are zero. Random weights break the symmetry between units — identical weights would mean identical gradients and units that never differentiate. Biases can start at zero since the weights now differ.

In training, both are updated by the same rule:

w ← w − η ∂L/∂w    b ← b − η ∂L/∂b

The gradient with respect to a weight includes the input value (∂z/∂w = x). So a feature that is always near zero gives near-zero gradients and its weight barely moves. The gradient with respect to a bias is just the upstream gradient (∂z/∂b = 1). So biases update no matter the input scale — another reason they train reliably.

One deliberate exception to zero-initialised biases: setting the final bias of a binary classifier to the log-odds of the positive class makes the net start out predicting the base rate. It noticeably speeds up early training on imbalanced data.

Count the params, then see what each one does

A layer is a matrix and a vector. This counts them for a real design, then moves single vals to watch what they control.

import numpy as np

def layer_shapes(sizes):
    return [(sizes[i], sizes[i + 1]) for i in range(len(sizes) - 1)]

arch = [784, 128, 64, 10]
print("a small classifier: %s\n" % " -> ".join(map(str, arch)))
print("%-14s %14s %10s %12s" % ("layer", "weight shape", "biases", "parameters"))
total = 0
for i, (a, b) in enumerate(layer_shapes(arch)):
    n = a * b + b
    total += n
    print("%-14s %14s %10d %12d" % ("layer %d" % (i + 1), "(%d, %d)" % (a, b), b, n))
print("%-14s %14s %10s %12d" % ("total", "", "", total))
print("\n%.1f%% of them are in the first layer -- parameters concentrate"
      % (100 * (784 * 128 + 128) / total))
print("where the input is widest.")

Now see what a single weight and bias each control when you nudge them.

import numpy as np

print("what one weight controls. a single neuron, two inputs:\n")
w = np.array([0.5, -1.2])
b = 0.3
x = np.array([2.0, 1.0])
print("  x = %s, w = %s, b = %.1f" % (x, w, b))
print("  z = w.x + b = %.2f*%.1f + %.2f*%.1f + %.1f = %.2f"
      % (w[0], x[0], w[1], x[1], b, x @ w + b))
print("\n  nudge each parameter by +0.1 and watch z:")
for i in range(2):
    w2 = w.copy(); w2[i] += 0.1
    print("    w[%d] +0.1 -> z %.2f   (changed by %+.2f = 0.1 * x[%d])"
          % (i, x @ w2 + b, (x @ w2 + b) - (x @ w + b), i))
print("    b    +0.1 -> z %.2f   (changed by +0.10, always)" % (x @ w + b + 0.1))
print("\n  a weight's influence is scaled by its input. the bias's is not.")
print("\nwithout a bias a layer is stuck passing through the origin:")
for xv in (0.0, 1.0, 2.0):
    print("    x=%.0f  w*x = %.2f   w*x + b = %.2f" % (xv, 2.0 * xv, 2.0 * xv + 3.0))
print("  at x=0 the first column can only be 0. no weight fixes that.\n")
rng = np.random.default_rng(0)
W = rng.normal(0, 0.1, (4, 3))
bb = np.zeros(3)
batch = rng.normal(size=(5, 4))
out = batch @ W + bb
print("the same layer as a framework stores it:")
print("  input %s @ weights %s + bias %s = output %s" % (batch.shape, W.shape, bb.shape, out.shape))
print("  the bias is one number per OUTPUT, broadcast across all rows.")
print("  %d weights against %d biases." % (W.size, bb.size))

Questions people ask

How many params does a layer have? inputs × units weights plus units biases.

Why initialise weights randomly but biases to zero? Random weights break symmetry between units. Once they differ, zero biases cause no problem.

Can weights be negative? Yes, and roughly half will be. A negative weight means the feature counts against that unit's activation.

Should biases be regularised? No. Weight decay is often applied to weights just — biases carry no capacity worth penalising.

What does a weight of zero mean? That the connection is effectively unused. L1 regularisation gives just zero weights. L2 gives small but non-zero ones.

Do biases increase capacity? Not in the sense of expressiveness across features. But they let each edge sit where the data needs rather than through the origin — which is a large real gap.

Recap in one screen

• Weights are the learned multipliers on connections. Biases are one learned offset per unit.
• Weights set the edge's way and strength. The bias sets where the threshold sits.
• Without a bias, each edge passes through the origin.
• Individual weights are interpretable in a one-layer net and unreliable in a deep one.
• Very uneven weight magnitudes often mean unscaled inputs, not an insight.
• Drop the bias when batch normalisation follows. And exclude biases from weight decay.