Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.
Each layer takes the previous layer's output and applies
a(l) = f(W(l) a(l−1) + b(l))
where W is a weight matrix, b is a bias vector, and f is the activation. Stack these and the network can represent increasingly complex functions — but only because f is non-linear. Remove it and the whole stack collapses into a single matrix.
A neural network is one operation repeated: multiply by a matrix, add a bias, bend the result. Depth is just how many times you do it.
Take 4 input features → one hidden layer of 8 → 3 output classes.
Add a second hidden layer of 8 and you insert another 8 × 8 + 8 = 72 — more than doubling the model. Parameter count grows with the product of adjacent layer widths, which is why wide layers next to each other get expensive fast.
A layer computes a weighted sum and applies an activation. Stack two linear layers and you have gained nothing at all: the composition of two matrix multiplications is another matrix multiplication, so a hundred linear layers collapse to a single equivalent one.
The activation is what breaks that collapse. Because it is non-linear, the composition cannot be flattened, and each additional layer can express something the previous one could not. This is the entire reason depth exists — remove the activations and a deep network is exactly as powerful as logistic regression, however many layers it has.
The universal approximation theorem says one sufficiently wide hidden layer can approximate any continuous function to arbitrary accuracy. That is reassuring and almost useless in practice, because “sufficiently wide” can mean exponentially many neurons.
Depth is what makes it tractable. Layers build features hierarchically: in a vision network the first layer responds to edges, the second to corners and textures assembled from those edges, the third to object parts, and so on. Each layer reuses the previous layer’s features rather than rediscovering them, so a deep network needs far fewer neurons than a shallow one to express the same function.
There is no formula, but there are reliable starting points. Begin with one or two hidden layers for tabular data; deeper networks rarely help there and images or sequences are better served by architectures designed for them. Set the width somewhere between the input and output sizes, and prefer a rough funnel — wider near the input, narrowing toward the output.
Then let the training curves decide. If the model cannot drive training loss down it lacks capacity, so widen or deepen it. If training loss is near zero while validation loss climbs, it has too much capacity for the data available, and the answer is regularisation or more data rather than a smaller network. Increase capacity until the model can overfit, then regularise back — that order is far more reliable than guessing an architecture up front.
A neural network is a chain of very simple operations. Each layer takes a vector of numbers, multiplies it by a matrix of weights, adds a bias, and pushes the result through a non-linear function.
a(1) = f(W(1)x + b(1)) → a(2) = f(W(2)a(1) + b(2)) → …
That is the whole forward pass. A network with 175 billion parameters is doing exactly this, many times.
The three parts of the architecture:
Remove the activation function and stack two layers: W₂(W₁x) = (W₂W₁)x. That is a single matrix. A hundred stacked linear layers can express exactly what one can — a straight boundary.
Insert a non-linear bend between them, and the layers stop collapsing. Each one can reshape the space the next one sees, so the network can represent curves, corners and disjoint regions.
This is the answer to "why not just use one very wide layer?" The universal approximation theorem says one hidden layer, made wide enough, can approximate any continuous function. In practice depth is dramatically more efficient: a function that needs an exponentially wide shallow network can often be represented by a narrow deep one, because layers compose features rather than enumerating them.
The layers build a hierarchy, and it is visible when you inspect trained networks.
| Depth | On images | On text |
|---|---|---|
| Layer 1 | Edges, colour blobs | Character and word shapes |
| Middle | Textures, parts | Phrases, syntax |
| Deep | Objects | Meaning, topic, intent |
This is why you cannot usefully read a single weight in a deep network: a feature's influence is spread across many paths, and the meaningful units are combinations rather than individual neurons.
It is also why transfer learning works. Early layers learn primitives that apply to almost any task in the domain, so they can be reused; only the last layers are specific to the original labels.
Repeat for every batch, for many epochs. Nothing else happens. Everything else in deep learning — optimisers, normalisation, schedules, regularisation — is a refinement of one of those four steps.
Depth without non-linearity. Stacking linear layers gains you nothing at all mathematically — it is the single most common misunderstanding about why deep networks work.A first hidden layer far wider than the input. Four inputs into a 512-unit layer adds thousands of parameters without adding any information; there were only ever four numbers to work with. Width is worth adding where there is signal to spread out, not at the front by default.
Multiply, add a bias, bend — repeated, with the bend being the part that matters.
| Decision | Typical answer |
|---|---|
| Hidden layers | Start with 2–3 for tabular data; use a proven architecture for images or text |
| Units per layer | 64–512, often narrowing towards the output |
| Hidden activation | ReLU, or GELU in transformers |
| Output activation | None for regression, softmax for multi-class, sigmoid for multi-label |
| Loss | MSE for regression, cross-entropy for classification |
| Optimiser | AdamW |
| Learning rate | 1e-3 for Adam, then tune — the most important hyperparameter |
| Batch size | 32–256, as large as memory allows |
| Regularisation | Weight decay, dropout, early stopping |
Two notes on that table. The output layer usually has no activation in modern frameworks, because the loss function applies softmax or sigmoid internally in a numerically stable way — applying it twice is a real and quiet bug. And the learning rate deserves its label: it affects results more than the number of layers, the width, or the choice of optimiser.
Deep learning is not the default answer to a modelling problem. It wins decisively on unstructured data — images, audio, text, video — where the raw input has spatial or sequential structure a network can exploit.
On tabular data, gradient-boosted trees usually match or beat a neural network while needing less tuning, less data and no GPU. This has been tested repeatedly and the result is consistent.
Three other reasons to look elsewhere: you have a few hundred rows, you need to explain each decision to a regulator, or you need the model to extrapolate outside the range it was trained on.
NaN; too low crawls. Find it with a short learning-rate sweep.Two layers, a forward pass, a backward pass and a training loop -- on a problem a linear model cannot touch. Everything a framework does is here, just written out.
How many layers should I use? For tabular data, two or three. For images and text, start from a proven architecture rather than designing one.
What is an epoch? One full pass through the training data. Batches are the chunks within it, and one update happens per batch.
Why does my loss go to NaN? Learning rate too high, a log(0) in the loss, or unscaled inputs. Check in that order.
Do I need a GPU? For images and text, effectively yes. For small tabular models, a CPU is fine.
How much data do I need? Thousands of examples per class as a rough floor, and far fewer if you fine-tune a pretrained model.
Why is it called a "neuron"? Historical analogy to biology. The resemblance is loose; a unit is a weighted sum with a threshold, not a cell.
Answer without scrolling back up.
What do the hidden layers of a network actually do?
Each layer re-describes its input in terms the next layer finds easier. That learned re-description is the thing deep learning buys you over hand-engineered features.
Widening a layer from 64 to 512 units mainly increases:
More parameters means more that can be memorised. Extra capacity with no extra data or regularisation is the standard route into overfitting.
All weights are initialised to exactly zero. What happens?
Identical weights get identical gradients, so the units never differentiate. Random initialisation exists to break exactly this symmetry.
where W is a weight matrix, b is a bias vector, and f is the activation. Stack these and the network can represent increasingly complex functions — but only because f is non-linear. Remove it and the whole stack collapses into a single matrix.