Weights & Biases

A concrete example

One input, weight w = 2.0, bias b = 0. At x = 0.5 you get z = 1.0 — the neuron fires.

Now set b = −1.5 and leave everything else alone: z = 1.0 − 1.5 = −0.5. Same input, same weight, opposite decision. The bias moved the threshold from x > 0 to x > 0.75.

This is why a bias is not optional detail. It is the difference between "is this input positive?" and "is this input above the level that actually matters?"

Why the bias cannot be dropped

Without a bias, the weighted sum is zero whenever every input is zero, so the decision boundary is forced through the origin. That is a severe restriction: a neuron that should fire only when its input exceeds 5 cannot express that threshold at all, because it has no way to shift its output independently of the inputs.

The bias supplies exactly that freedom. In y = wx + b, w tilts the line and b slides it — and the same is true in a thousand dimensions, where the weights orient a hyperplane and the bias offsets it from the origin. One extra parameter per neuron buys the ability to place the boundary anywhere rather than only through a single fixed point.

Counting a layer's parameters

A fully connected layer with n_in inputs and n_out outputs has one weight per connection plus one bias per output neuron:

parameters = (n_in × n_out) + n_out

For 784 inputs and 128 outputs that is 784 × 128 + 128 = 100,480. The biases are 128 of them — about 0.1% of the total, which is why they are cheap enough to always include and why regularisation normally skips them.

Notice the weight count is a product. Doubling either the input or the output size doubles the parameters; doubling both quadruples them. That quadratic growth is why wide layers dominate a network’s memory footprint.

What training actually changes

Training changes nothing about the network except these numbers. The architecture, the activations and the connections are all fixed before the first step; gradient descent only ever adjusts weights and biases.

That is worth holding onto, because it clarifies what a trained model is: a specific set of values for these parameters. Saving a model saves the weights and biases. Transfer learning reuses another model’s weights and biases. A randomly initialised network and a fully trained one differ in no other respect.

One concrete example

One input, weight w = 2.0, bias b = 0. At x = 0.5 you get z = 1.0 — the neuron fires.

Now set b = −1.5 and change nothing else: z = 1.0 − 1.5 = −0.5. Same input, same weight, opposite decision.

The bias moved the threshold. Before, the neuron fired whenever x > 0; now it fires whenever x > 0.75. That is the difference between asking "is this input positive?" and "is this input above the level that actually matters?"

Which is why a bias is not an optional detail. A network without biases can only draw boundaries through the origin, and almost no real decision boundary passes through the origin.

What they are, and how many there are

Weights are the multipliers on the connections between units. A layer taking 100 inputs into 50 units has a 100×50 weight matrix — 5,000 numbers, one for every input-output pair.

Biases are one number per unit — 50 in that layer — added after the weighted sum.

z = w · x + b

Both are learned. Everything else you set — the number of layers, the learning rate, the batch size — is in service of finding good values for these two.

A useful proportion: biases are a tiny fraction of a network's parameters. A layer with 100 inputs and 50 units has 5,000 weights and 50 biases, so about 1%. They are cheap and they matter.

Reading trained weights, and the limits of doing so

In a single-layer model, a weight's magnitude is interpretable: a large positive coefficient on "smoking" means smoking pushes the prediction up. That reading is why logistic regression survives in regulated settings.

In anything deeper, it breaks down. A feature's influence is spread across many paths, and a small first-layer weight can still matter enormously downstream through amplification in later layers. Reading a single weight as feature importance in a deep network is unreliable, and integrated gradients or SHAP are the tools that do the job properly.

Two things that are worth reading from trained weights:

Wildly different magnitudes across features is almost always a symptom of unscaled inputs, not a finding. If one feature is measured in thousands and another in decimals, the network compensates with tiny and huge weights, and gradient descent struggles to move both usefully. Scale the inputs instead.

The distribution of weight magnitudes tells you whether regularisation is doing anything. A histogram concentrated near zero with a few large values is what weight decay produces.

Try this above

1. Set Number of Inputs to 1 so there is a single weight to watch.
2. Change that weight and watch the response curve steepen and flip direction. Negative weights invert the neuron's opinion of that feature.
3. Now change only the bias. The shape stays identical; it slides sideways.
4. Switch Activation Function between Step and Sigmoid and repeat — the geometry is the same, only the sharpness changes.

What usually goes wrong

Wildly different weight magnitudes are almost always a symptom of unscaled inputs rather than a real finding. If one feature is measured in thousands and another in decimals, the network compensates with tiny and huge weights, and gradient descent struggles to move both usefully. Scale the inputs instead.Reading a single weight as feature importance is unreliable in anything deeper than one layer. In a multi-layer network a feature's influence is spread across many paths, and a small first-layer weight can still matter enormously downstream.

In one line

Weights decide direction and strength; the bias decides where the threshold sits.

Where the bias is left out on purpose

There are two standard cases where a bias is omitted.

Before batch normalisation. Normalisation subtracts the mean, which cancels any constant the previous layer added. So a convolution followed by batch normalisation has a redundant bias, and reference implementations write bias=False. It saves parameters and changes nothing.

In some attention projections, where recent language models drop biases from the query/key/value projections and sometimes from the feed-forward layers as well — they turn out to contribute little at scale, and removing them simplifies the model slightly.

Otherwise, keep the bias. The default in every framework includes it for good reason.

Their role during training

At initialisation, weights are small random values and biases are zero. Random weights break the symmetry between units — identical weights would mean identical gradients and units that never differentiate. Biases can start at zero because the weights already differ.

During training, both are updated by the same rule:

w ← w − η ∂L/∂w    b ← b − η ∂L/∂b

The gradient with respect to a weight includes the input value (∂z/∂w = x), so a feature that is always near zero produces near-zero gradients and its weight barely moves. The gradient with respect to a bias is just the upstream gradient (∂z/∂b = 1), so biases update regardless of the input scale — another reason they train reliably.

One deliberate exception to zero-initialised biases: setting the final bias of a binary classifier to the log-odds of the positive class makes the model start out predicting the base rate, which noticeably speeds up early training on imbalanced data.

Count the parameters, then see what each one does

A layer is a matrix and a vector. This counts them for a real architecture, then moves single values to watch what they control.

import numpy as np

def layer_shapes(sizes):
    return [(sizes[i], sizes[i + 1]) for i in range(len(sizes) - 1)]

arch = [784, 128, 64, 10]
print("a small classifier: %s" % " -> ".join(map(str, arch)))
print()
print("%-14s %14s %10s %12s" % ("layer", "weight shape", "biases", "parameters"))
total = 0
for i, (a, b) in enumerate(layer_shapes(arch)):
    n = a * b + b
    total += n
    print("%-14s %14s %10d %12d" % ("layer %d" % (i + 1), "(%d, %d)" % (a, b), b, n))
print("%-14s %14s %10s %12d" % ("total", "", "", total))
print()
print("%.1f%% of them are in the first layer. that is the usual shape --"
      % (100 * (784 * 128 + 128) / total))
print("parameters concentrate where the input is widest.")
print()

print("what one weight controls. a single neuron, two inputs:")
w = np.array([0.5, -1.2])
b = 0.3
x = np.array([2.0, 1.0])
print("  x = %s, w = %s, b = %.1f" % (x, w, b))
print("  z = w.x + b = %.2f*%.1f + %.2f*%.1f + %.1f = %.2f"
      % (w[0], x[0], w[1], x[1], b, x @ w + b))
print()
print("  nudge each parameter by +0.1 and watch z:")
for i in range(2):
    w2 = w.copy(); w2[i] += 0.1
    print("    w[%d] +0.1 -> z %.2f   (changed by %+.2f = 0.1 * x[%d])"
          % (i, x @ w2 + b, (x @ w2 + b) - (x @ w + b), i))
print("    b    +0.1 -> z %.2f   (changed by +0.10, always)" % (x @ w + b + 0.1))
print()
print("  a weight's influence is scaled by its input. the bias's is not --")
print("  that is the whole difference between them, and it is why the bias")
print("  can shift the whole function while a weight only tilts it.")
print()

print("without a bias a layer is stuck passing through the origin:")
for xv in (0.0, 1.0, 2.0):
    print("    x=%.0f  w*x = %.2f   w*x + b = %.2f" % (xv, 2.0 * xv, 2.0 * xv + 3.0))
print("  at x=0 the first column can only be 0. no weight fixes that.")
print()

print("the same layer, written the way a framework stores it:")
rng = np.random.default_rng(0)
W = rng.normal(0, 0.1, (4, 3))     # 4 inputs -> 3 outputs
bb = np.zeros(3)
batch = rng.normal(size=(5, 4))     # 5 rows at once
out = batch @ W + bb
print("  input  %s @ weights %s + bias %s = output %s"
      % (batch.shape, W.shape, bb.shape, out.shape))
print("  the bias is one number per OUTPUT, broadcast across all 5 rows.")
print("  that broadcast is why the shapes work and why a bias is cheap:")
print("  %d weights against %d biases." % (W.size, bb.size))

Questions people ask

How many parameters does a layer have? inputs × units weights plus units biases.

Why initialise weights randomly but biases to zero? Random weights break symmetry between units; once they differ, zero biases cause no problem.

Can weights be negative? Yes, and roughly half will be. A negative weight means the feature counts against that unit's activation.

Should biases be regularised? No. Weight decay is normally applied to weights only — biases carry no capacity worth penalising.

What does a weight of zero mean? That the connection is effectively unused. L1 regularisation produces exactly zero weights; L2 produces small but non-zero ones.

Do biases increase capacity? Not in the sense of expressiveness across features, but they let every boundary sit where the data requires rather than through the origin — which is a large practical difference.

Recap in one screen

• Weights are the learned multipliers on connections; biases are one learned offset per unit.
• Weights set the boundary's direction and strength; the bias sets where the threshold sits.
• Without a bias, every boundary passes through the origin.
• Individual weights are interpretable in a one-layer model and unreliable in a deep one.
• Very uneven weight magnitudes usually mean unscaled inputs, not an insight.
• Drop the bias when batch normalisation follows, and exclude biases from weight decay.