A convolution with no spatial extent sounds pointless. It is one of the most useful layers in modern architectures.
Overview
The operation that looks like nothing
A 3×3 convolution mixes a pixel with its eight neighbours. A 1×1 convolution has no neighbours to combine. At first reading it appears to times each pixel by a number. This Is a scaling and hardly worth a layer.
That reading forgets the channel dim, and the channel dim is where all happens.
A feature map isn't a grid of numbers. It's a stack of grids, one per channel. A layer with 256 channels holds 256 values at each spatial spot. A 1×1 convolution stands at one spot, takes all 256 values there, and finds a weighted sum of them. Then does it again for each output channel, and then repeats the whole thing at each spot with the same weights.
So it doesn't mix neighbours. It mixes channels. Drag the two controls above and watch the link pattern: each input channel reaches each output channel, and nothing spatial happens at all.
1x1 Convolutions
This module needs JavaScript: the
images are computed in the page rather than downloaded.
Worth knowing
A 1×1 convolution looks at one pixel position and all of its channels. It mixes channels, never neighbours.
It is a fully-connected layer applied identically at every spatial position, which is why it is written as a convolution at all.
Its main job is changing the channel count — usually reducing it before an expensive layer, which is what a bottleneck is.
Weights: Cin × Cout, against 9 × Cin × Cout for a 3×3.
1x1 Convolutions
A kernel with no spatial extent. Why almost each modern architecture is full of them.
It is a fully-connected layer in disguise
Fix one spatial spot and the op is just a dense layer: Cin inputs, Cout outputs, Cin × Cout weights. What makes it a convolution is that the same dense layer is applied, unchanged, at each spot in the map.
That is the same weight-sharing arg that motivates convolution in the first place. If mixing channels in a given way is handy at one location, it's probably handy at all of them, and sharing the weights makes the layer free of the input's spatial size.
What it is actually for
Changing the channel count. This is the common case. A 1×1 convolution is the cheapest possible way to turn 256 channels into 64, or 64 into 256, and it's why the layer is at times called a map.
Bottlenecks. Put a channel reduction before an costly spatial convolution and an expansion after it. ResNet's bottleneck block is just this: 1×1 down to 64 channels, 3×3 at 64, 1×1 back up to 256. The 3×3 — by far the costliest part — runs on a quarter of the channels, and the two 1×1s cost almost nothing by check.
Count it. A 3×3 straight from 256 to 256 channels needs 9 × 256 × 256 ≈ 590,000 weights. The bottleneck version needs 256×64 + 9×64×64 + 64×256 ≈ 70,000. Same input and output shape, an eighth of the params, and an extra two non-linearities thrown in.
Adding non-linearity without touching detail. Each 1×1 is followed by an activation, so a stack of them grows representational depth at constant spatial size and negligible cost. This was the "network in network" idea that named the technique.
Replacing the classifier head. Global mean pooling followed by a 1×1 convolution does the job of a large dense layer with a fraction of the params, and works for any input size.
The arithmetic
The readout above checks the two counts right. For Cin input and Cout output channels:
Kernel
Weights
Sees
1×1
Cin × Cout
one position, all channels
3×3
9 × Cin × Cout
3×3 positions, all channels
Depthwise 3×3
9 × Cin
3×3 positions, one channel each
That last row is worth noticing. A depthwise separable convolution splits the work in two: a depthwise 3×3 that mixes neighbours but not channels, followed by a 1×1 that mixes channels but not neighbours. Together they rough a full 3×3 at roughly a ninth of the cost. MobileNet is built almost fully from that pair, and half of it's 1×1 convolutions.
The bottleneck, counted
The 1×1's main job is changing the channel count cheaply, and the ResNet bottleneck shows why that is worth a layer. Compare a straight 3×3 from 256 to 256 channels against the bottleneck that squeezes down, does the spatial work narrow, and expands back:
design
weights
one 3×3, 256 → 256
9 × 256 × 256 = 589,824
1×1 to 64, 3×3 at 64, 1×1 to 256
256·64 + 9·64·64 + 64·256 = 69,632
Same input and output shape, 8.5× fewer params, and two extra non-linearities thrown in — because the costly 3×3 now runs on a quarter of the channels while the two 1×1s that resize the stack cost almost nothing. That is the whole reason deep ResNets are affordable. The 1×1 is doing no spatial work at all. Fix one spot and it's just a dense layer, Cin × Cout weights, applied unchanged at each spot — so it adds nothing to the receptive field. Its entire value is mixing channels. This Is just what the spatial kernels can't afford to do at full width.
Beyond resizing channels, the 1×1 buys non-linearity at constant detail — each is followed by an activation. So A stack of them deepens the network spatially for almost nothing. This Was the "network in network" idea that named the technique — and it's half of a depthwise-separable convolution, the pointwise step that mixes channels after the depthwise step mixes neighbours. Two cautions: the bottleneck is a genuine info bottleneck. So Squeezing 256 channels to 8 before the spatial work can cost more fit than the work it saves. And a 1×1 with no activation after it, stacked on another linear layer, merges into one linear map, because two mats times'd together are a single mat.
Where it goes wrong with 1x1 Convolutions
Expecting spatial work from it. It can't smooth, sharpen or find an edge. If the receptive field needs to grow, a 1×1 adds nothing — its part to the receptive field is just zero.
Reducing channels too aggressively. The bottleneck is a genuine info bottleneck. Squeezing 256 channels to 8 before the spatial convolution saves work and can cost more fit than it's worth.
Losing the activation. A 1×1 with no non-linearity after it, stacked on another linear layer, merges into a single linear map. Two mats times'd together are one mat.
Questions people ask about 1x1 Convolutions
What does a 1x1 convolution do? It takes a weighted sum across channels at each pixel independently. Spatial neighbours are not mixed at all.
Why use one if it sees no neighbours? To change the number of channels cheaply and add a non-linearity. It reduces or expands channel depth without touching spatial size.
How is it different from a fully connected layer? It is the same weights applied at every position, so it works on any image size. A fully connected layer is tied to a fixed input shape.
What is a bottleneck block? A 1x1 reduces channels, a 3x3 does the spatial work on the thinner tensor, and another 1x1 expands again. This saves many multiplications, as in ResNet.
How many parameters does a 1x1 convolution have? Input channels times output channels, plus biases. Going from 256 to 64 channels takes 16,384 weights.
Where did the idea come from? From the Network in Network paper, and it was popularised by Inception modules for dimension reduction.
A convolution that looks at one pixel
A 1x1 kernel has no spatial extent at all. This Sounds useless. It's a per-pixel linear layer across channels. It's what makes bottleneck blocks and depthwise-separable convolutions affordable.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
H, W, C = 4, 4, 6
x = rng.normal(0, 1.0, (H, W, C))
print("a feature map: %dx%d positions, %d channels each." % (H, W, C))
print(" the vector at position (0,0): %s" % np.round(x[0, 0], 3))
print()
COUT = 3
W1 = rng.normal(0, 0.4, (C, COUT))
out = x @ W1
print("a 1x1 convolution with %d output channels is a %dx%d matrix applied"
% (COUT, C, COUT))
print("to every position independently:")
print(" output at (0,0): %s" % np.round(out[0, 0], 3))
print(" by hand: %s" % np.round(x[0, 0] @ W1, 3))
print(" shape %s -> %s" % (x.shape, out.shape))
print(" the spatial dimensions are untouched. only the channel count")
print(" changed, which is the whole purpose.")
print()
print("IT IS A DENSE LAYER, SHARED ACROSS POSITIONS. proof -- shuffle the")
print("positions and the outputs follow exactly:")
flat = x.reshape(-1, C)
perm = rng.permutation(H * W)
a = (flat @ W1)[perm]
b = flat[perm] @ W1
print(" identical: %s" % np.allclose(a, b))
print(" so it cannot mix information between positions. a 3x3 convolution")
print(" can; this cannot. it only mixes CHANNELS.")
Output
The two main practical uses are cutting parameter counts and building bottleneck blocks.
example_02.pyNumPy
import numpy as np
print("USE 1 -- CHANGING THE CHANNEL COUNT CHEAPLY. compare the cost of")
print("reducing 256 channels to 64:")
for kh in (1, 3, 5):
p = kh * kh * 256 * 64
print(" %dx%d conv, 256->64 : %12s parameters" % (kh, kh, "{:,}".format(p)))
print(" the 1x1 does the same job for a ninth of a 3x3's cost, because")
print(" there is no spatial window to weight.")
print()
print("USE 2 -- THE BOTTLENECK BLOCK, which is what ResNet-50 is built from.")
print("a 3x3 convolution on 256 channels, done two ways:")
direct = 3 * 3 * 256 * 256
print(" direct 3x3, 256->256 : %12s"
% "{:,}".format(direct))
squeeze = 1 * 1 * 256 * 64
middle = 3 * 3 * 64 * 64
expand = 1 * 1 * 64 * 256
total = squeeze + middle + expand
print(" 1x1 256->64 (squeeze) : %12s" % "{:,}".format(squeeze))
print(" 3x3 64->64 (the actual spatial work) : %12s" % "{:,}".format(middle))
print(" 1x1 64->256 (expand) : %12s" % "{:,}".format(expand))
print(" %-40s: %12s" % ("bottleneck total", "{:,}".format(total)))
print(" %.1fx fewer parameters, and the expensive 3x3 now runs on 64"
% (direct / total))
print(" channels instead of 256 -- a %.0fx saving on that layer alone."
% ((3 * 3 * 256 * 256) / (3 * 3 * 64 * 64)))
print()
print(" and it is still non-linear: there is a ReLU after each of the")
print(" three, so the block is not equivalent to one big convolution.")
Output
A 1x1 convolution also serves as a cross-channel mixer and turns a classifier into a fully convolutional segmenter.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
C, COUT = 6, 3
W1 = rng.normal(0, 0.4, (C, COUT))
_ = rng.normal(0, 1.0, (4, 4, C)) # advance RNG to match earlier state
_ = rng.permutation(16)
print("USE 3 -- CROSS-CHANNEL MIXING. each output channel is a learned")
print("combination of all input channels. here is what one output channel")
print("is made of:")
for j in range(COUT):
contrib = np.abs(W1[:, j])
order = np.argsort(-contrib)
print(" output channel %d draws most on input channels %s"
% (j, ", ".join("%d (%.2f)" % (i, W1[i, j]) for i in order[:3])))
print(" a 3x3 convolution also mixes channels, but it spends 9 weights per")
print(" pair doing it. if the mixing is all you need, 8 of those 9 are")
print(" wasted.")
print()
print("USE 4 -- MAKING A NETWORK FULLY CONVOLUTIONAL. a dense layer needs a")
print("fixed input size; a 1x1 convolution does not:")
for size in (7, 14, 28):
feat = rng.normal(size=(size, size, 512))
logits = feat @ rng.normal(0, 0.1, (512, 10))
print(" %2dx%-2d input -> logits %s -- one prediction PER POSITION"
% (size, size, logits.shape))
print(" that is how a classifier becomes a segmenter: replace the final")
print(" dense layer with a 1x1 convolution and it outputs a class map")
print(" instead of a class, at whatever resolution you feed it.")
print()
print("the summary in one line: a 1x1 convolution is the cheapest way to")
print("change how many channels you have, and the only convolution that")
print("does nothing spatial at all.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does a 1x1 convolution actually combine?
It has no spatial extent, so it never touches neighbours. At each position it takes a weighted sum across the whole channel stack, and repeats that with shared weights everywhere.
Why does a ResNet bottleneck put 1x1 convolutions around the 3x3?
The 3x3 cost scales with Cin x Cout. Reducing to 64 channels first and expanding after cuts a 590,000-weight block to about 70,000 for the same input and output shape.
What happens to two stacked 1x1 convolutions with no activation between them?
Each is a matrix multiplication at every position. Two matrices multiplied together are one matrix, so the pair has exactly the representational power of one layer.
Cheat sheet
1x1 Convolutions
See how a 1x1 convolution mixes channels without touching spatial dimensions, and why it appears in Inception, ResNet and every bottleneck.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.