Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.
Each layer takes the last layer's output and applies
a(l) = f(W(l) a(l−1) + b(l))
where W is a weight matrix, b is a bias vector. And f is the activation. Stack these and the network can represent increasingly complex functions — but just since f is non-linear. Remove it and the whole stack collapses into a single matrix.
The weight matrix W connects each neuron in one layer to each neuron in the next. Each entry is one weight. And each row computes one neuron's input as a dot product of the weights with the last layer's output. The bias vector shifts the result before the activation, letting the neuron to fire still when all inputs are zero.
Training adjusts the weights and biases to minimise a loss function. The algorithm is gradient descent: compute the loss on a batch of data, propagate the gradient of that loss backward through the network to find how each weight added. And nudge each weight in the way that reduces the loss. That backward pass is backpropagation. And it is a direct application of the chain rule from calculus.
The design — how many layers, how many neurons per layer. It activation functions — is chosen before training. More layers let the network compose more complex features. But they also increase the count of params, the risk of overfitting. And the trouble of training. The real starting point for most problems is one or two hidden layers with ReLU activations and an output layer whose activation matches the task: sigmoid for binary classification, softmax for multiclass. And no activation for regression.
The count of params finds the net's capacity and its memory footprint. Each weight and each bias is one param, stored as a floating-point count. A network with 10 million params uses about 40 MB at 32-bit precision. Knowing how to count params from the design is the first step in grasp whether a net is appropriately sized for the data.
Interactive architecture builder.
A neural network is one operation repeated: multiply by a matrix, add a bias, bend the result. Depth is just how many times you do it.
Each layer takes the last layer's output and applies
a(l) = f(W(l) a(l−1) + b(l))
where W is a weight matrix, b is a bias vector. And f is the activation. Stack these and the network can represent increasingly complex functions — but just since f is non-linear. Remove it and the whole stack collapses into a single matrix.
The weight matrix W connects each neuron in one layer to each neuron in the next. Each entry is one weight. And each row computes one neuron's input as a dot product of the weights with the last layer's output. The bias vector shifts the result before the activation, letting the neuron to fire still when all inputs are zero.
Training adjusts the weights and biases to minimise a loss function. The algorithm is gradient descent: compute the loss on a batch of data, propagate the gradient of that loss backward through the network to find how each weight added. And nudge each weight in the way that reduces the loss. That backward pass is backpropagation. And it is a direct application of the chain rule from calculus.
The design — how many layers, how many neurons per layer. It activation functions — is chosen before training. More layers let the network compose more complex features. But they also increase the count of params, the risk of overfitting. And the trouble of training. The real starting point for most problems is one or two hidden layers with ReLU activations and an output layer whose activation matches the task: sigmoid for binary classification, softmax for multiclass. And no activation for regression.
The count of params finds the net's capacity and its memory footprint. Each weight and each bias is one param, stored as a floating-point count. A network with 10 million params uses about 40 MB at 32-bit precision. Knowing how to count params from the design is the first step in grasp whether a net is appropriately sized for the data.
Take 4 input features → one hidden layer of 8 → 3 output classes.
Add a second hidden layer of 8 and you insert another 8 × 8 + 8 = 72 — more than doubling the net. Param count grows with the product of next layer widths. It is why wide layers next to each other get costly fast.
A layer computes a weighted sum and applies an activation. Stack two linear layers and you have gained nothing at all: the composition of two matrix multiplications is another matrix multiplication. So a hundred linear layers collapse to a single equivalent one.
The activation is what breaks that collapse. Since it is non-linear, the composition cannot be flattened. And each extra layer can express something the last one could not. This is the whole reason depth exists — remove the activations and a deep network is just as powerful as logistic regression, however many layers it has.
The universal approximation theorem says one sufficiently wide hidden layer can approximate any continuous function to arbitrary accuracy. That is reassuring and almost useless in practice, since “sufficiently wide” can mean exponentially many neurons.
Depth is what makes it tractable. Layers build features hierarchically: in a vision network the first layer responds to edges, the second to corners and textures assembled from those edges, the third to object parts. And so on. Each layer reuses the last layer’s features rather than rediscovering them. So a deep network needs far fewer neurons than a shallow one to express the same function.
There is no formula, but there are reliable starting points. Begin with one or two hidden layers for tabular data. Deeper networks rarely help there and pics or sequences are better served by designs designed for them. Set the width somewhere between the input and output sizes. And prefer a rough funnel — wider near the input, narrowing toward the output.
Then let the training curves decide. If the net cannot drive training loss down it lacks capacity. So widen or deepen it. If training loss is near zero while validation loss climbs, it has too much capacity for the data on hand. And the answer is regularisation or more data rather than a less network. Increase capacity until the net can overfit, then regularise back — that order is far more reliable than guessing an design up front.
A neural network is a chain of very simple steps. Each layer takes a vector of counts, multiplies it by a matrix of weights, adds a bias. And pushes the result through a non-linear function.
a(1) = f(W(1)x + b(1)) → a(2) = f(W(2)a(1) + b(2)) → …
That is the whole forward pass. A network with 175 billion params is doing just this, many times.
The three parts of the design:
Remove the activation function and stack two layers: W₂(W₁x) = (W₂W₁)x. That is a single matrix. A hundred stacked linear layers can express just what one can — a straight edge.
Insert a non-linear bend between them, and the layers stop collapsing. Each one can reshape the space the next one sees. So the network can represent curves, corners and disjoint regions.
This is the answer to "why not just use one very wide layer?" The universal approximation theorem says one hidden layer, made wide enough, can approximate any continuous function. In practice depth is dramatically more fast: a function that needs an exponentially wide shallow network can often be shown by a narrow deep one, since layers compose features rather than enumerating them.
The layers build a hierarchy. And it is visible when you inspect trained networks.
| Depth | On images | On text |
|---|---|---|
| Layer 1 | Edges, colour blobs | Character and word shapes |
| Middle | Textures, parts | Phrases, syntax |
| Deep | Objects | Meaning, topic, intent |
This is why you cannot usefully read a single weight in a deep network: a feature's influence is spread across many paths. And the meaningful units are combinations rather than individual neurons.
It is also why transfer learning works. Early layers learn primitives that apply to almost any task in the domain. So they can be reused. Just the last layers are set to the first labels.
Repeat for each batch, for many epochs. Nothing else happens. All else in deep learning — optimisers, normalisation, schedules, regularisation — is a refinement of one of those four steps.
Depth without non-linearity. Stacking linear layers gains you nothing at all mathematically — it is the single most common misunderstanding about why deep networks work.A first hidden layer far wider than the input. Four inputs into a 512-unit layer adds thousands of params without adding any information. There were just ever four counts to work with. Width is worth adding where there is signal to spread out, not at the front by default.
Multiply, add a bias, bend — repeated, with the bend being the part that matters.
| Decision | Typical answer |
|---|---|
| Hidden layers | Start with 2–3 for tabular data; use a proven architecture for images or text |
| Units per layer | 64–512, often narrowing towards the output |
| Hidden activation | ReLU, or GELU in transformers |
| Output activation | None for regression, softmax for multi-class, sigmoid for multi-label |
| Loss | MSE for regression, cross-entropy for classification |
| Optimiser | AdamW |
| Learning rate | 1e-3 for Adam, then tune — the most important hyperparameter |
| Batch size | 32–256, as large as memory allows |
| Regularisation | Weight decay, dropout, early stopping |
Two notes on that table. The output layer often has no activation in modern frameworks, since the loss function applies softmax or sigmoid internally in a numerically stable way — applying it twice is a real and quiet bug. And the learning rate deserves its label: it affects results more than the count of layers, the width. Or the choice of optimiser.
Deep learning is not the default answer to a modelling problem. It wins decisively on unstructured data — pics, audio, text, video — where the raw input has spatial or sequential structure a network can exploit.
On tabular data, gradient-boosted trees often match or beat a neural network while needing less tuning, less data and no GPU. This has been tested repeatedly and the result is consistent.
Three other reasons to look elsewhere: you have a few hundred rows, you need to explain each decision to a regulator. Or you need the net to extrapolate outside the range it was trained on.
NaN. Too low crawls. Find it with a short learning-rate sweep.Two layers, a forward pass, a backward pass and a training loop -- on a problem a linear net cannot touch. All a framework does is here, just written out.
Building on the same setup, the next block shows the left cases.
The next block covers the left steps.
How many layers should I use? For tabular data, two or three. For pics and text, start from a proven design rather than designing one.
What is an epoch? One full pass through the training data. Batches are the chunks within it, and one update happens per batch.
Why does my loss go to NaN? Learning rate too high, a log(0) in the loss. Or unscaled inputs. Check in that order.
Do I need a GPU? For pics and text, effectively yes. For small tabular nets, a CPU is fine.
How much data do I need? Thousands of cases per class as a rough floor. And far fewer if you fine-tune a pretrained net.
Why is it called a "neuron"? Historical analogy to biology. The resemblance is loose. A unit is a weighted sum with a threshold, not a cell.
Answer without scrolling back up.
What do the hidden layers of a network actually do?
Each layer re-describes its input in terms the next layer finds easier. That learned re-description is the thing deep learning buys you over hand-engineered features.
Widening a layer from 64 to 512 units mainly increases:
More parameters means more that can be memorised. Extra capacity with no extra data or regularisation is the standard route into overfitting.
All weights are initialised to exactly zero. What happens?
Identical weights get identical gradients, so the units never differentiate. Random initialisation exists to break exactly this symmetry.
Build a neural network layer by layer and watch data flow forward through weights, biases and activations into a prediction.