The whole design, in one rule
VGG-16 is worth studying not because it is fast — it is not — but because it proved that a simple, repeated structure can beat a complicated one. Almost every later architecture borrows the idea of stacking small filters instead of using large ones.
Before VGG, convolutional architectures were a grab bag of kernel sizes: AlexNet opened with an 11×11 stride-4 convolution, followed by 5×5s and then 3×3s. Simonyan and Zisserman asked what happens if you fix the kernel at the smallest size that still has a notion of direction — 3×3, stride 1, padding 1. And get depth by stacking rather than by widening.
That single rule makes the entire network. Spatial size never changes inside a block, because padding 1 on a 3×3 kernel is just size-preserving. It halves only at the five max-pools. Channels double at each pool, from 64 to 512, and then stop.
| Block | Convolutions | Channels | Output at 224 |
|---|
| 1 | 2 | 64 | 112 × 112 |
| 2 | 2 | 128 | 56 × 56 |
| 3 | 3 | 256 | 28 × 28 |
| 4 | 3 | 512 | 14 × 14 |
| 5 | 3 | 512 | 7 × 7 |
Thirteen convolutions, five pools, then flatten 7×7×512 = 25,088 values into three fully-connected layers of 4096, 4096 and 1000.
Why two 3x3s beat one 5x5
This is the arg the paper is really about, and it's worth doing in numbers.
A single 5×5 convolution with C input and C output channels has 25C² params. Two stacked 3×3 convolutions have 9C² + 9C² = 18C² — 28% fewer — and their receptive field is the same: each output of the second layer depends on a 5×5 patch of the input.
Three 3×3s reach 7×7 with 27C² params against 49C², a 45% saving. And the stack has two ReLUs inside it where the single large kernel has none, so the same receptive field is now found by a more expressive function rather than one linear map.
This is the arg that ended large kernels in vision for a decade. It's also the arg that later got in part reversed — ConvNeXt and the modern large-kernel networks reopened it. But on the terms VGG set: depth of small kernels is the default, and any of it else needs a reason.
Where the parameters actually are
Push the layer table in the explorer to the bottom and the numbers stop being reasonable:
| Layer | Parameters | Share |
|---|
| All 13 convolutions | 14,714,688 | 10.6% |
| fc6 (25088 → 4096) | 102,764,544 | 74.3% |
| fc7 (4096 → 4096) | 16,781,312 | 12.1% |
| fc8 (4096 → 1000) | 4,097,000 | 3.0% |
| Total | 138,357,544 | |
One layer is three quarters of the model. fc6 takes the flattened 7×7×512 feature map and connects each one of its 25,088 values to each of 4096 outputs. There's no weight sharing and no locality. It's a dense mat with a hundred million entries, applied once per image.
Now flip to the math view. The convolutions do about 15.3 billion times-builds up per image and the dense layers about 0.12 billion. So:
- the convolutions are 11% of the params and 99% of the find;
- the dense layers are 89% of the params and 1% of the find.
That inversion is the single most handy thing to take from this page. The param count tells you about memory, download size and overfitting risk. The MAC count tells you about latency and energy. They aren't the same number, they aren't even linked here, and quoting one when you meant the other is a routine source of confusion.
What one 3x3 layer actually costs
Take conv3-64 in block 1, at 224×224. Its kernel is 3×3, it reads 64 channels and writes 64:
parameters = 64 x 64 x 3 x 3 + 64 = 36,928
positions = 224 x 224 = 50,176
MACs = 36,864 x 50,176 = 1,849,688,064
One point eight billion times-builds up from thirty-seven thousand numbers. The weights are reused at each spot. This Is the entire point of a convolution and the reason the math and the storage are so far apart.
Now the same sum for fc6:
parameters = 25,088 x 4,096 + 4,096 = 102,764,544
positions = 1
MACs = 102,760,448
A hundred million params used just once each. Selecting any conv layer in the explorer prints its own version of this math. Selecting a dense one prints the other kind. Doing it once by hand is what makes the two budgets stop feeling like the same number.
What replaced the head, and why
Turn on Swap the FC head for global mean pooling in the explorer. The param count falls from 138 M to about 15 M — a 9× reduction — and nothing else about the network changes.
Global mean pooling merges each of the 512 final feature maps to its own mean, giving 512 numbers with no params at all, and a single 512→1000 layer finishes the job. GoogLeNet did this in the same year VGG was published; ResNet did it the year after. It has been standard ever since.
It buys three things beyond size. It drops the layer most prone to overfitting. It makes the network accept any input detail, because the pool doesn't care how large the map it's blending is — whereas fc6 demands just 25,088 inputs. This Is why the first VGG only accepts 224×224. And it forces each final feature map to match to a thing class-relevant on its own. This Is what makes class activation maps work.
Move the input detail slider with the FC head on, and watch the flattened size — and so fc6's param count — change with it. That dependency is the reason detail was frozen.
Reading the shape of a network from its table
The layer table is worth reading as a shape rather than a list, because the same shape recurs in almost each convolutional network built since.
Follow two columns down it. The spatial size goes 224, 112, 56, 28, 14, 7 — halving five times. The channel count goes 64, 128, 256, 512, 512 — doubling four times and then stopping. Times them: the tensor at the first block holds 64 × 112² = 802,816 values, and at the last 512 × 7² = 25,088. The form shrinks by a factor of 32 as it goes.
That is the whole compression story of a classifier. The input is 150,528 numbers showing colour at spots. The output is 1,000 numbers showing belief about categories. each block trades some spatial detail for some semantic width, and the halve-and-double rule is the exchange rate.
The rule isn't any. Halving both spatial dims quarters the number of spots, and doubling the width quadruples the per-spot cost of a convolution, so the math per block stays roughly constant while the info gets steadily more abstract. Move the detail slider and watch the MAC column: the cost scales with the square of the input size, which is why detail is the most effective single lever on inference latency, and why anyone optimising a vision model reaches for it before they reach for a smaller architecture.
The two questions to ask of any architecture
VGG is the clearest place to learn a habit that pays off on each model after it: read the param budget and the math budget as separate questions, and ask which layers dominate each.
Params answer "how large is the file, how much GPU memory does it occupy, and how much data will it take to fit". Times-builds up answer "how long does one image take and how much energy does it cost". A layer can dominate one and be invisible in the other, and VGG's fc6 is the extreme case: three quarters of the model and under one per cent of the work.
The inversion runs the other way too. A 3×3 convolution over 64 channels has 36,928 params — nothing — and applies each one of them at 112×112 spots, which is 462 million times-builds up for a layer you wouldn't notice in a param table.
Once you've the habit, the optimisation advice stops being folklore. If the model is too large to ship, look at the widest dense layers and the last stages. If it's too slow, look at the input detail and the early stages. Those are different problems with different fixes, and confusing them is why people prune a network for weeks and find it runs at just the same speed.
VGG's real legacy
As a classifier VGG is obsolete: ResNet-50 gets better ImageNet fit with a fifth of the params and a quarter of the math. No one should train one today.
It lives for a different reason. Because the architecture is so plain — no residuals, no branches, no norming, just a stack of the same convolutions — its mid activations are unusually well behaved, and a visual loss found on VGG features is still the standard choice in style transfer, super-detail and image generation. When a paper says it optimises "VGG loss" or "LPIPS with a VGG backbone", this is the network being used, often only up to relu3_3 or relu4_3, with the hundred-million-param head discarded fully.
import torch
import torchvision
vgg = torchvision.models.vgg16(weights="IMAGENET1K_V1")
# The 13 convolutions and 5 pools, without the 123 M-parameter classifier.
features = vgg.features[:16].eval() # up to relu3_3
for p in features.parameters():
p.requires_grad_(False)
def perceptual_loss(x, y):
return torch.nn.functional.l1_loss(features(x), features(y))
Two details matter in that snippet and both bite people. Slicing .features rather than using the whole model is what drops the dense head, and freezing the params is what keeps it a *loss* rather than a second network being trained. And what you feed it must be normed with the ImageNet mean and standard deviation the weights were trained with, or the features are being read off a spread the network has never seen.
Questions people ask
Why is the input frozen at 224×224? Because fc6 is a dense mat expecting just 25,088 inputs. This Is 7×7×512. And 7×7 is what 224 turns after five halvings. Feed it 256×256 and the flatten makes 8×8×512 = 32,768 values. This Doesn't fit the mat. Global mean pooling drops the constraint fully, which is why each architecture after VGG accepts any detail.
VGG-16 or VGG-19 — does the difference matter? Rarely. VGG-19 adds one convolution to each of the last three blocks and about 20 M params, for a few tenths of a point of ImageNet fit. For visual loss the 16-layer version is the conventional choice and the one most published numbers assume.
Which param count is right, 138 M or 134 M? Both get quoted. 138,357,544 includes the 1000-way fc8. Smaller figures often come from a model with the classifier removed or replaced. Check what a count includes before checking two of them.
Why does it use so much GPU memory for its size? Because activations, not weights, dominate training memory. The first block holds 64×224×224 values per image. 3.2 M numbers. And each one is kept for the backward pass, while the 102 M-param fc6 stores 4,096. That is why VGG batch sizes are small though ResNet-50 is deeper.
Is relu3_3 or relu4_3 the right layer for visual loss? Earlier layers reward matching texture and colour. Later ones reward matching structure and handle texture differences. Style transfer often takes a few at once, super-detail one of those two. There's no correct answer, only a choice about what you want preserved.
Should I fine-tune VGG for a new task? You can. It will work. You're paying 138 M params for fit ResNet-50 exceeds at 25.6 M. Reach for VGG when you in detail want its features. A loss, a published check. Not as a default backbone.
Recap in one screen
- One rule makes the network: 3×3, stride 1, padding 1, with the spatial size changing only at the five max-pools.
- Two stacked 3×3s match a 5×5's receptive field for 18C² params against 25C². Add a non-linearity in between.
- The convolutions are 11% of the params and 99% of the math. The dense head is 89% and 1%. Read the two budgets as separate questions.
- fc6 alone is 102 M params used once per image. Global mean pooling replaces it, cuts the model about 9×, and frees the input detail.
- Obsolete as a classifier, still standard as a visual-loss backbone. Sliced before the head, frozen, and fed ImageNet-normed input.