Visualize how a flattened 1D array of features connects to every node in the output layer using weights, biases, and Softmax to make final classifications.
Dense Network Graph
COMPUTING...
Dot Product Formula
Zj = Σ (Xi · Wij) + Bj
Click "Forward Pass" to view calculations.
Overview
The Transition to 1D
Throughout the early stages of a CNN, pics are handled as 2D grids (or 3D volumes with color channels). Convolutional and pooling layers extract spatial features like edges, textures, and shapes. However, to make a final classification (e.g., "Is this a cat or a dog?"), the network must flatten this 2D/3D data into a single 1D list of counts. This is the input vector you see on the left.
The dense layer (also called a fully connected layer) then takes this 1D vector and connects each input to each output through a learned weight. If the flattened vector has 512 items and the dense layer has 128 neurons, there are 512 × 128 = 65,536 weights plus 128 biases. Each neuron computes a weighted sum of the whole input, adds its bias. And passes the result through an activation function — often ReLU in hidden layers and softmax in the final classification layer.
The convolutional layers before it are spatial: each filter looks at a small patch and shares weights across the pic. The dense layer is the opposite: it sees the whole feature vector at once, with no weight sharing. It is how it mixes features from other spatial locations into a global decision. An "eye" feature in the top-left and an "ear" feature in the top-right can just be mixed into "face" by a layer that sees both at once.
Modern designs often replace the dense classifier head with global average pooling. It averages each feature map into a single count. This reduces the param count drastically (no dense weights at all) and acts as a structural regulariser. ResNet, EfficientNet and most recent designs use this way, keeping a single dense layer just for the final class logits.
Visualize how a flattened 1D array of features connects to every node in the output layer using weights, biases, and Softmax to make final classifications.
The decision-making powerhouse at the end of a Convolutional Neural Network.
The Transition to 1D
Throughout the early stages of a CNN, pics are handled as 2D grids (or 3D volumes with color channels). Convolutional and pooling layers extract spatial features like edges, textures, and shapes. However, to make a final classification (e.g., "Is this a cat or a dog?"), the network must flatten this 2D/3D data into a single 1D list of counts. This is the input vector you see on the left.
The dense layer (also called a fully connected layer) then takes this 1D vector and connects each input to each output through a learned weight. If the flattened vector has 512 items and the dense layer has 128 neurons, there are 512 × 128 = 65,536 weights plus 128 biases. Each neuron computes a weighted sum of the whole input, adds its bias. And passes the result through an activation function — often ReLU in hidden layers and softmax in the final classification layer.
The convolutional layers before it are spatial: each filter looks at a small patch and shares weights across the pic. The dense layer is the opposite: it sees the whole feature vector at once, with no weight sharing. It is how it mixes features from other spatial locations into a global decision. An "eye" feature in the top-left and an "ear" feature in the top-right can just be mixed into "face" by a layer that sees both at once.
Modern designs often replace the dense classifier head with global average pooling. It averages each feature map into a single count. This reduces the param count drastically (no dense weights at all) and acts as a structural regulariser. ResNet, EfficientNet and most recent designs use this way, keeping a single dense layer just for the final class logits.
Why "Fully Connected"?
It is called a Fully Connected (or Dense) layer since each single node in the input vector is connected to each single node in the output vector. If you have 4 inputs and 3 outputs, there are $4 \times 3 = 12$ distinct connections (weights), plus 3 biases.
Each weight represents the "weight" of a set feature for a set class. For case, if Input Feature 2 strongly represents "pointy ears", the weight connecting Input 2 to the "Cat" output node will likely be a high positive count, while its connection to "Bird" might be negative.
Where the convolutions stop and the decision starts
Convolutional layers give feature maps: a stack of grids saying where each learned pattern was found. That is not yet an answer to "which of these ten classes is it?"
The dense (fully connected) layer is what turns the evidence into a decision. Each input value connects to each output unit, with its own weight:
outputj = Σi (wij × inputi) + bj
Since each input touches each output, a dense layer can mix evidence from anywhere in the pic at once — something a convolution, with its small local window, deliberately cannot do. "Whiskers in the upper left and a tail in the less right" is a combination just a global layer can express right.
The final dense layer has one unit per class. And its outputs are logits — unbounded scores that a softmax turns into probabilities.
Getting from a 3-D tensor to a vector
Convolutions give something like 7×7×512. A dense layer needs a flat vector. Two ways to get there. And the choice matters more than it looks.
Flatten lays the whole tensor out in order: 7×7×512 = 25,088 vals. A dense layer of 4,096 units on top of that holds 25,088 × 4,096 ≈ 103 million params — in one layer.
Global average pooling reduces each feature map to its mean: 512 vals, one per channel. A dense layer to 1,000 classes then needs about 513,000 params.
Two hundred times fewer, and it often generalises better. Three reasons it won:
Far fewer params, so far less overfitting.
No dependence on input size, since the count of channels does not change with resolution.
Each output corresponds to "how much of this feature was present total". It is right interpretable.
VGG used flatten and carried 138 million params, most of them in the classifier head. ResNet uses global average pooling and has 25 million in total, while being both deeper and more exact. That single change is one of the clearest architectural improvements in the field's history.
Dense layers as 1×1 convolutions
There is a useful equivalence: a dense layer applied at each spatial position is just a 1×1 convolution.
That is what makes fully convolutional networks possible. Replace the dense head of a classifier with 1×1 convolutions and the network accepts any input size, giving a coarse map of class scores rather than a single vector — which is where segmentation designs began.
It also explains why 1×1 convolutions appear all over modern designs: they mix information across channels at each position without touching spatial structure. It is just a per-pixel dense layer. ResNet's bottleneck blocks use them to shrink and restore channel counts cheaply.
Real guidance
Prefer global average pooling to flatten. It is the modern default for good reasons.
One hidden dense layer is often plenty after good convolutional features. Two is at times justified, three almost never.
Dropout belongs here. Dense layers hold most of a naive network's params and overfit first. 0.5 dropout between dense layers was standard practice for just this reason.
The output layer has no activation in most frameworks — the loss function applies softmax internally. And applying it twice is a real and quiet bug.
Check the flattened size when changing input resolution. A hard-coded dense input size is the most common shape error when adapting an design.
Guided Experiments with This Interactive
Observe the Forward Pass:
Click the Forward Pass button. Watch the center canvas. The network computes the output one node at a time. It highlights the connections, multiplies each input by its corresponding weight, sums them up. And adds a bias. This raw sum is called the Logit ($Z$).
Tweak the Weights:
Look at the Weights table on the right. Change the weights connecting to the "Cat" column to high positive counts (e.g., 2.0). Click Forward Pass again. Notice how the raw score (Logit) for the Cat class shoots up!
The Role of Bias:
At the bottom of the weights table are the Biases. A bias is an extra value added to the final sum. It shifts the activation function left or right. Try setting the Bird bias to a high value like 5.0. It will artificially inflate the Bird's probability no matter the inputs.
Softmax Activation:
Raw Logits can be any count (e.g., -5.2, 0.4, 12.8). To interpret them as probabilities, we apply the Softmax function. Softmax squashes the logits into vals between 0 and 1, ensuring they all sum up to just 1.0 (or 100%). Notice how the progress bars in the Output section always balance each other out.
The Dense Formula
For an output node $j$, the raw Logit ($Z_j$) is the dot product of the inputs ($X$) and weights ($W$), plus bias ($b$):
Zj = (X1·W1j + X2·W2j + ...) + bj
Softmax is then applied to turn $Z$ into probabilities ($P$):
Pj = eZj / Σ(eZ)
The layer that finally forgets where things were
After the convolutions comes a dense layer. And it is the first part of the network with no notion of position at all. This builds one by hand, shows just what flatten does to the geometry. And measures the param blow-up that made everyone stop with them.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(8)
C, H, W = 4, 3, 3 # a tiny final feature map
fmap = np.round(rng.uniform(0, 1, (C, H, W)), 2)
print("THE FINAL FEATURE MAP: %d channels of %dx%d = %d numbers."
% (C, H, W, C * H * W))
for c in range(C):
print(" channel %d" % c)
for row in fmap[c]:
print(" " + " ".join("%5.2f" % v for v in row))
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
print("STEP 1 -- FLATTEN. the same numbers, in a line:")
flat = fmap.reshape(-1)
print(" " + " ".join("%5.2f" % v for v in flat))
print(" nothing was computed. no parameters, no arithmetic -- flatten is")
print(" purely a change of shape.")
print()
print(" but look at what it did to the NEIGHBOURS. in the map, cell")
print(" (0,0,0) sits beside (0,0,1) and above (0,1,0). after flatten:")
def idx(c, r, k):
return c * H * W + r * W + k
print("%-26s %10s %10s" % ("cell", "flat index", "distance"))
base = idx(0, 0, 0)
for label, (c, r, k) in (("(0,0,0) itself", (0, 0, 0)),
("(0,0,1) its right neighbour", (0, 0, 1)),
("(0,1,0) the cell below it", (0, 1, 0)),
("(1,0,0) same place, next channel", (1, 0, 0))):
print("%-34s %6d %10d" % (label, idx(c, r, k), abs(idx(c, r, k) - base)))
print(" the right neighbour is 1 away, the cell BELOW is %d away, and the"
% W)
print(" same position in the next channel is %d away. the 2D adjacency"
% (H * W))
print(" that every convolution depended on is now an arbitrary numbering.")
print(" the dense layer will not recover it -- and does not try to.")
print()
Output
The next block covers the left steps.
example_03.pyNumPy
import numpy as np
print("STEP 2 -- THE DENSE LAYER: out = W @ x + b. that is the whole thing.")
NOUT = 3
Wm = np.round(rng.normal(0, 0.4, (NOUT, flat.size)), 2)
bias = np.round(rng.normal(0, 0.1, NOUT), 2)
out = Wm @ flat + bias
print(" weight matrix: %s bias: %s" % (str(Wm.shape), str(bias.shape)))
print(" every output touches every input. output 0, term by term:")
terms = Wm[0] * flat
print(" " + " ".join("%.2f*%.2f" % (Wm[0, i], flat[i])
for i in range(5)) + " ...")
print(" sum of all %d terms: %.4f, plus bias %.2f = %.4f"
% (flat.size, terms.sum(), bias[0], out[0]))
print(" outputs: " + " ".join("%.4f" % v for v in out))
print()
print("STEP 3 -- WHY POSITION IS GONE. move the whole feature map one cell")
print("to the right and re-run the SAME dense layer:")
shifted = np.zeros_like(fmap)
shifted[:, :, 1:] = fmap[:, :, :-1]
out_shift = Wm @ shifted.reshape(-1) + bias
print("%-22s %s" % ("original", " ".join("%8.4f" % v for v in out)))
print("%-22s %s" % ("shifted by 1", " ".join("%8.4f" % v for v in out_shift)))
print("%-22s %s" % ("difference",
" ".join("%8.4f" % v for v in (out_shift - out))))
print(" completely different answers for a picture of the same thing")
print(" moved one pixel. a convolution would have produced the same")
print(" feature map, shifted. the dense layer has no such property and")
print(" cannot acquire one -- it would have to learn every weight")
print(" pattern again at every offset.")
print()
Output
The next block continues with the left demonstrations.
example_04.pyNumPy
import numpy as np
print("STEP 4 -- THE COST. dense layers are where the parameters go:")
print("%-34s %16s %16s" % ("layer", "parameters", "share"))
conv_params = [("conv 3x3, 3->64", 3 * 64 * 9 + 64),
("conv 3x3, 64->128", 64 * 128 * 9 + 128),
("conv 3x3, 128->256", 128 * 256 * 9 + 256)]
dense_params = [("flatten(256*7*7) -> 4096", 256 * 7 * 7 * 4096 + 4096),
("dense 4096 -> 4096", 4096 * 4096 + 4096),
("dense 4096 -> 1000", 4096 * 1000 + 1000)]
total = sum(n for _, n in conv_params + dense_params)
for name, n in conv_params + dense_params:
print("%-34s %16s %15.1f%%" % (name, "{:,}".format(n), 100.0 * n / total))
print("%-34s %16s" % ("total", "{:,}".format(total)))
cn = sum(n for _, n in conv_params)
dn = sum(n for _, n in dense_params)
print(" convolutions: %s (%.1f%%). dense: %s (%.1f%%)."
% ("{:,}".format(cn), 100.0 * cn / total,
"{:,}".format(dn), 100.0 * dn / total))
print(" the first dense layer alone is %.0fx the size of all three"
% (dense_params[0][1] / cn))
print(" convolutions combined, and it is doing far less: one matrix")
print(" multiply, no weight sharing, no reuse of any parameter anywhere.")
Output
Building on the same setup, the next block shows the left cases.
example_05.pyNumPy
print()
print("WHY IT IS SO EXPENSIVE: a convolution reuses EVERY ONE of its")
print("weights at all %d positions of the map. a dense layer uses each" % (7 * 7))
print("weight exactly once. that is the entire difference, and it explains")
print("both")
print("the parameter count and the loss of translation invariance -- they")
print("are the same fact stated twice.")
print()
print("%-28s %-24s %s" % ("", "convolution", "dense"))
for row in (("weights per output", "k*k*Cin", "the whole input"),
("reused across positions", "yes", "no"),
("shift the input", "output shifts", "output changes"),
("input size must be fixed", "no", "yes"),
("knows about geometry", "yes", "no")):
print("%-28s %-24s %s" % row)
print()
print("modern classifiers keep exactly ONE dense layer, right at the end,")
print("after global average pooling has already reduced the map to one")
print("number per channel. at that point the input is %d numbers instead" % C)
print("of %d, the parameter count is trivial, and there is nothing spatial" % (C * H * W))
print("left to destroy -- the dense layer's job is only to turn a list of")
print("'what was found' into a list of class scores.")
Output
Summing up
Dense layers sit at the very end of a CNN to do classification.
They need 1D flattened input vectors.
Each input is connected to each output, creating a massive count of weight params.
Softmax transforms raw dot-product scores (Logits) into readable probabilities.
What each part adds
Component
Job
Convolution layers
Detect patterns, locally and repeatedly
Pooling / stride
Reduce size, widen the view
Global average pooling
Summarise each feature map into one number
Dense hidden layer
Combine features globally
Dropout
Prevent the dense layers from memorising
Output dense layer
One logit per class
Softmax (in the loss)
Turn logits into probabilities
Reading a trained network's dense weights is at times informative: a large positive weight from a particular channel to a particular class means that feature is evidence for that class. With global average pooling this is unusually interpretable. And it is just the mechanism Class Activation Mapping exploits to give heatmaps showing where the evidence came from.
Why not use dense layers throughout?
The natural question, given that dense layers are more general than convolutions. Three answers.
Params. A dense first layer on a 224×224×3 pic needs 150 million weights per 1,000 units. A convolution needs a few thousand.
No spatial prior. A dense layer treats neighbouring pixels as unrelated. So it has to learn from data that they are tied — which needs far more data.
No translation equivariance. A feature learned in one corner does not transfer to another. So each position needs its own cases.
Convolutions win not since they are more powerful but since they are more constrained. And the constraint matches how pics really work. That is what an inductive bias is. And it is why CNNs still beat vision transformers on small datasets. There the transformer has to learn the same structure from scratch.
Questions people ask
How many units should the hidden dense layer have? Somewhere between the feature count and the class count — 128 to 512 is typical. Bigger rarely helps once the convolutional features are good.
Do I need a hidden dense layer at all? Often not. Modern designs often go straight from global average pooling to the output layer.
Why is my net overfitting badly? Check whether a flatten-and-dense head holds most of the params. Replacing it with global average pooling often fixes it outright.
Should I apply softmax in the net? No, in PyTorch — CrossEntropyLoss expects logits. In Keras, from_logits=True does the same thing. Applying softmax twice trains slowly and quietly badly.
What is the gap between dense and linear? Nothing — Keras calls it Dense, PyTorch calls it Linear.
Can dense layers handle var input sizes? Not right. Global average pooling before them removes the constraint.
Recap in one screen
Dense layers connect all to all. It is what lets a global decision from local features.
Flatten-then-dense is where a naive CNN's params explode. Global average pooling replaces it at a fraction of the cost.
A dense layer applied per position is just a 1×1 convolution — the basis of fully convolutional networks.
Put dropout between dense layers. Leave the output layer unactivated and let the loss apply softmax.
Convolutions beat dense layers on pics since their constraints match how pics are structured.
A worked size math
Tracing shapes through a small network makes the dense layer's cost concrete. Input 224×224×3:
Stage
Output shape
Values
After block 1 (stride 2)
112×112×64
802,816
After block 2
56×56×128
401,408
After block 3
28×28×256
200,704
After block 4
14×14×512
100,352
After block 5
7×7×512
25,088
Flatten
25,088
25,088
Global average pool
512
512
Both final rows describe the same tensor, summarised differently. A dense layer to 1,000 classes costs 25 million params from the flatten and half a million from the pooled version.
That is the whole argument. And it also explains a common failure when adapting an design: change the input from 224 to 256 and the flattened size changes from 25,088 to 32,768. So a hard-coded dense layer errors out. With global average pooling the size is 512 either way. And nothing breaks.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Transition to 1D”?
What does this module say about “Why "Fully Connected"”?
What does this module say about “Where the convolutions stop and the decision starts”?
Cheat sheet
Fully Connected Layer in CNN
Flatten a CNN's feature maps into a 1D vector and feed it through a dense layer that mixes spatial features into a class decision.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.