The question the module answers
each convolutional layer forces a choice: 1×1, 3×3 or 5×5? The right answer depends on how large the thing you're looking for is in that layer's units. This Varies by image, by class and by depth. VGG answers it by fiat — always 3×3 — and gets the range of scales from depth.
Inception refuses the question. Run 1×1, 3×3, 5×5 and a pooling branch in parallel on the same input, and concatenate the results along the channel axis. The next layer then has all the scales available and learns which to weight.
This only works because each branch is padded to preserve spatial size. A 1×1 with no padding, a 3×3 with padding 1 and a 5×5 with padding 2 all leave a 28×28 map as 28×28, so their outputs stack cleanly. Concatenation along channels is or else hopeless: you can't concatenate a 28×28 map with a 26×26 one.
The naive version is unaffordable
Take module 3a, the first in GoogLeNet: 192 input channels at 28×28, making 64 + 128 + 32 channels from the three convolutional branches plus what the pool passes through.
Turn the bottleneck off in the explorer and read the branch costs:
| Branch | Multiply-accumulates |
|---|
| 1×1 → 64 | 9.6 M |
| 3×3 → 128, reading all 192 channels | 173.4 M |
| 5×5 → 32, reading all 192 channels | 120.4 M |
| Total | 303 M |
Three hundred million times-builds up for one module, and there are nine of them. Worse, the pooling branch passes its 192 input channels straight through, so the output has 64 + 128 + 32 + 192 = 416 channels — more than went in. Stack two of these and the third one is reading 416 channels, then the fourth is reading even more. The channel count grows without bound and the cost grows with the *square* of it.
The 1x1 convolution is the fix
A 1×1 convolution has no spatial extent at all. At each pixel it takes the vector of C input channels and applies a single learned mat to give C' outputs. It's a per-pixel linear map across channels — a change of basis in channel space, applied the same way all over. And its cost is C × C' per pixel.
Put one in front of each costly branch and the math merges:
| Branch | Naive | With a reduction |
|---|
| 3×3 | 192→128 directly: 173.4 M | 192→96 then 96→128: 101.1 M |
| 5×5 | 192→32 directly: 120.4 M | 192→16 then 16→32: 12.4 M |
| pool | 192 passed through, 0 M | 192→32 projection: 4.8 M |
The 5×5 branch drops by a factor of ten. A 5×5 kernel over 16 channels is a twelfth of the work of the same kernel over 192. And the pool branch now *shrinks* its channel count instead of growing it, so the module outputs 256 channels rather than 416 and the next module has less to read.
Total: 128 M against 303 M, and the output is smaller. Toggle the control and watch both numbers move together.
The whole network
The full GoogLeNet stacks nine of these modules with two mid pools, after a conventional 7×7 and 3×3 stem. The layer table in the explorer lists all nine with their real widths from table 1 of the paper.
Two things stand out. The widths aren't a pattern. Module 4d puts 288 channels in its 3×3 branch and 64 in its 5×5, while 4e puts 320 and 128 — because they were tuned rather than derived. And the whole network comes to 6.8 million params against VGG-16's 138 million, on about 1.5 billion times-builds up against VGG's 15.5 billion. It won ILSVRC 2014 classification. VGG came second.
The head is the other half of that story. GoogLeNet ends with global mean pooling and one 1024→1000 layer, not with two 4096-wide dense layers. That choice alone accounts for most of the 20× param gap.
The first also carried two auxiliary classifiers — extra softmax heads hanging off the middle of the network during training, their losses added at weight 0.3 — to push grad into the early layers of a network too deep to train or else. They were removed at inference. Within a year ResNet solved that problem properly with identity shortcuts, and later Inception versions dropped the auxiliary heads. The paper's own follow-up concluded they acted more as regularisers than as grad highways.
What the family did next
- Inception-v2/v3 factorised the 5×5 into two 3×3s, then went further and factorised an n×n into an n×1 followed by a 1×n. It added batch norming and label smoothing.
- Inception-v4 and Inception-ResNet added residual links, which sped up training a lot without changing the fit ceiling much — a handy data point about what residuals really buy.
- Xception took the factorisation to its limit: if a 1×1 handles cross-channel mixing and the spatial kernel handles space, do them fully apart. That is depthwise separable convolution, and it's the basis of MobileNet and of most fast architectures since.
import torch
import torch.nn as nn
class InceptionModule(nn.Module):
def __init__(self, cin, c1, r3, c3, r5, c5, pp):
super().__init__()
self.b1 = nn.Conv2d(cin, c1, 1)
self.b2 = nn.Sequential(nn.Conv2d(cin, r3, 1), nn.ReLU(inplace=True),
nn.Conv2d(r3, c3, 3, padding=1))
self.b3 = nn.Sequential(nn.Conv2d(cin, r5, 1), nn.ReLU(inplace=True),
nn.Conv2d(r5, c5, 5, padding=2))
self.b4 = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1),
nn.Conv2d(cin, pp, 1))
def forward(self, x):
# Padding is chosen per branch so all four leave the map the same size.
return torch.cat([self.b1(x), self.b2(x), self.b3(x), self.b4(x)], dim=1)
m = InceptionModule(192, 64, 96, 128, 16, 32, 32) # module 3a
print(sum(p.numel() for p in m.parameters())) # 163,696
Note the padding values: 0, 1 and 2 for kernels 1, 3 and 5. Get one of them wrong and torch.cat raises a shape error on dim=1 — which is the architecture telling you that the branches have to agree spatially, and is worth triggering once on purpose.
What it costs to run four branches
There's a cost to the parallel structure that the times-build up count doesn't show, and it's worth naming because it shows why inception-style modules fell out of fashion despite being fast on paper.
Four branches means four separate convolution kernels, four separate memory allocations for their outputs, and a concatenation that has to gather them. On a GPU, a single large convolution saturates the hardware. Four small ones launched in sequence each spend part of their time not doing math at all. The measured latency of an inception module is always worse than its MAC count predicts, and the gap widens as hardware gets faster, because the fixed per-launch cost doesn't shrink.
This is the general lesson about speed metrics. Params, MACs and latency are three different budgets, and an architecture can win on the first two and lose on the third. Depthwise separable convolutions have the same problem in a more extreme form: they cut MACs by roughly eight or nine times and often deliver two or three times the speed, because they're memory-bandwidth bound rather than math bound.
The hands-on rule that follows: measure latency on the hardware you'll deploy to, and treat MAC counts as a rough guide rather than a guess. Two networks with the same MAC counts can differ by a factor of three in milliseconds.
The idea to keep
The 1×1 convolution is the transferable part. It isn't a special case of a convolution so much as a distinct tool: it changes the channel count at a cost linear in channels, without touching space. Once you can do that cheaply, costly ops become affordable by sandwiching them between a map down and a map back up. ResNet's bottleneck block is just that pattern. So is MobileNet's inverted residual. So Is the feed-forward block of a transformer, and so is each "reduce, operate, expand" structure you'll meet from here on.
Questions people ask
Why does a 1×1 convolution do any of it at all? Because it isn't spatial. It's a learned mat applied to the channel vector at each pixel. With 192 inputs and 96 outputs it's a 192×96 change of basis performed the same way all over. It can discard, combine or duplicate channels. It only looks trivial if you picture convolution as a thing that happens in space.
Where do the per-branch channel counts come from? From tuning, not a formula. Table 1 of the paper lists them per module and they aren't a pattern. 4d puts 288 channels in its 3×3 branch and 64 in its 5×5. This is the part of Inception that didn't generalise. The reason later architectures preferred one repeating block with a width multiplier.
Is GoogLeNet the same thing as Inception-v1? Yes. GoogLeNet is the exact 22-layer ILSVRC 2014 entry. Inception is the module and the family. Inception-v3 is the version most people really load. It expects 299×299 input rather than 224.
Why did the auxiliary classifiers go away? They existed to push grad into the early layers of a network too deep to train or else. Batch norming and then residual links solved that properly. The paper's own follow-up concluded the auxiliary heads had acted as regularisers rather than grad highways. Nothing replaced them because nothing needed to.
If it has 20× fewer params than VGG, why is it not 20× faster? Because params aren't latency. Four branches mean four kernel launches, four output allocations and a gather. The fixed cost per launch doesn't shrink as hardware gets faster. So measured latency is always worse than the MAC count predicts. Depthwise separable convolutions have the same problem more severely.
Should I use an Inception model today? For new work, no. A ResNet or a modern fast network is easier to train and better supported. Inception-v3 still matters in one place, though. The Inception Score and FID are defined on its activations. It remains load-bearing infrastructure for generative-model test.
Recap in one screen
- The module runs 1×1, 3×3, 5×5 and a pool branch in parallel and concatenates along channels. The next layer picks the scale.
- Concatenation needs each branch to preserve spatial size. This Is what the per-branch padding of 0, 1 and 2 is for.
- A 1×1 reduction in front of the costly branches cuts module 3a from 303 M to 128 M MACs *and* shrinks its output from 416 to 256 channels.
- The pool branch's map is what stops the channel count growing without bound as modules are stacked.
- GoogLeNet is 6.8 M params against VGG's 138 M, mostly because it ends in global mean pooling. The transferable idea is reduce, operate, expand.