Split a convolution into the part that mixes neighbours and the part that mixes channels. The saving is roughly a factor of nine.
Overview
What a full convolution is doing
A 3×3 convolution from 64 channels to 128 does two jobs at once. At each output spot it looks at a 3×3 spatial zone, and it mixes all 64 input channels. Each of the 128 output channels needs its own set of weights for all of that, so the count is:
k * k * Cin * Cout = 3 * 3 * 64 * 128 = 73,728
The question that leads to separable convolutions is whether those two jobs have to be done together.
Depthwise Separable Convolution
This module needs JavaScript: the
images are computed in the page rather than downloaded.
Worth knowing
A full convolution mixes neighbours and channels in one step, which is why it costs k² · Cin · Cout.
The depthwise step runs one k×k filter per input channel, mixing neighbours only.
The pointwise step is a 1×1 convolution, mixing channels only.
Together they cost k²·Cin + Cin·Cout, which for a 3×3 with many channels is about a ninth.
Depthwise Separable Convolution
One convolution split into two. Why almost each model that runs on a phone is built from them.
Doing them one at a time
Depthwise. Run one k×k filter per input channel, and keep the channels apart. Channel 7 is convolved with filter 7 and makes channel 7 of the output. Neighbours are mixed. Channels aren't. Cost: k * k * Cin, so 9 * 64 = 576.
Pointwise. Follow it with a 1×1 convolution, which mixes all the channels at each spot and has no spatial extent at all. Channels are mixed. Neighbours aren't. Cost: Cin * Cout, so 64 * 128 = 8,192.
Together: 8,768 against 73,728 for the same input and output shape. Drag the controls and the ratio in the readout stays close to 8–9×. Wherever you put them.
When Cout is large the 1/Cout term nearly vanishes and the ratio approaches k^2 — nine for a 3×3. Set the kernel control to 5 or 7 and the saving rises toward 25 and 49. This Is why separable convolutions matter more the larger the kernel.
Set output channels to 8 while leaving input at 64 and the ratio falls, because with few output channels the pointwise step is no longer cheap scaled to the whole. The technique earns its keep in wide layers. This Is where the cost was.
The cost of the saving
It isn't free, and pretending or else is the usual mistake.
A full convolution can learn any function of a 3×3×Cin zone. The separable pair can't: it's restricted to functions that factor into a spatial part and a channel part. That is a strictly smaller family, and on the same architecture a separable model often reaches slightly lower fit per layer.
What makes it worth doing is that the saving is much larger than the loss. With eight times fewer params you can afford more layers, wider layers, or a model that fits on the device at all. And the resulting network often beats the full-convolution one at equal cost.
Where it is used
MobileNet is built almost fully from these pairs, and was the paper that made the technique standard for on-device vision.
Xception applied the same arg to Inception, arguing that Inception modules were already approximating a separable convolution and that the extreme version worked better.
EfficientNet uses inverted residual blocks that expand channels with a 1×1, do the spatial work depthwise, and project back down — the same breakdown with an expansion around it.
A note on speed
Param count and wall-clock time aren't the same thing. A depthwise convolution does very little math per byte of memory it touches, so it's memory-bound, and hardware optimised for dense mat times doesn't reach any of it like its peak throughput on one.
An eight-times param reduction is so often two or three times faster rather than eight. Still worth having, and worth measuring rather than given.
Where the saving comes from, and how big
Splitting a convolution into a depthwise step (mix neighbours, keep channels apart) and a pointwise 1×1 (mix channels, skip neighbours) costs a fraction of doing both at once. The ratio is 1 / (1/Cout + 1/k²), which approaches k² as the output grows wide:
kernel
separable saving (Cout=128)
approaches
3×3
8.4×
9
5×5
20.9×
25
7×7
35.4×
49
So a 3×3 to 128 channels drops from 73,728 weights to 8,768, and the larger the kernel the more it saves. This Is why separable convolutions matter most in wide layers with big kernels. The cost is representational: the separable pair can only learn functions that factor into a spatial part and a channel part, a strictly smaller family than a full convolution. So It reaches slightly lower fit per layer — but the saving is so much larger than the loss that the freed budget buys more or wider layers that win total. And the param saving doesn't translate one-to-one into speed: a depthwise convolution does little math per byte it touches, so it's memory-bound, and an eight-times param cut is often only two or three times faster .
The technique earns its keep in wide layers. This Is where the cost was, and it underpins the on-device architectures: MobileNet is built almost fully from these pairs, Xception argued Inception modules were already approximating them and pushed the idea to its limit, and EfficientNet wraps them in inverted residual blocks that expand channels with a 1×1, do the spatial work depthwise, and project back down. Two failure modes are worth naming. Using a separable convolution in the first layer saves almost nothing — three input channels leave little to factor — and costs fit where the model can least afford it. And the depthwise and pointwise steps need a non-linearity between them, or the pair merges toward a single linear map rather than the two-stage breakdown it's meant to be.
Where it goes wrong with Depthwise Separable Convolution
Expecting the speed-up to match the param count. Measure it.
Using it in the first layer. With three input channels there's almost nothing to save, and the restriction costs fit where the model can least afford it.
Losing the non-linearity. The depthwise and pointwise steps need an activation between them, or the pair is closer to a single linear map than the two-stage breakdown it's supposed to be.
Questions people ask about Depthwise Separable Convolution
What does depthwise separable convolution do? It splits a standard convolution into a per-channel spatial filter followed by a 1x1 convolution that mixes channels. The result is similar in capacity at a fraction of the cost.
How much cheaper is it? The cost ratio is about 1/N + 1/K squared, for N output channels and K by K kernels. With 3x3 kernels and many channels that works out to roughly 8 to 9 times fewer operations.
Does it lose accuracy? Slightly, since it is a restricted form of convolution. Models like MobileNet and Xception show the loss is small compared with the savings.
Why is it not always faster in practice? Depthwise layers do little arithmetic per byte of memory moved, so they are often memory-bound. Real speedups are smaller than the operation count suggests.
Which models use it? MobileNet, Xception and EfficientNet all rely on it. It is the standard building block when the model must run on phones or embedded hardware.
What is the difference between depthwise and pointwise? Depthwise filters each channel on its own spatially. Pointwise is the 1x1 convolution that combines channels at each pixel.
Splitting one convolution into two cheaper ones
A normal convolution mixes space and channels at once. Doing those apart costs a fraction as much. This Is the whole idea behind MobileNet -- and this measures both the saving and what it gives up.
example_01.pyNumPy
import numpy as np
def standard(kh, cin, cout): return kh * kh * cin * cout
def depthwise(kh, cin): return kh * kh * cin
def pointwise(cin, cout): return cin * cout
print("a standard convolution does two jobs in one operation:")
print(" 1. combine a spatial neighbourhood (the kxk window)")
print(" 2. combine the channels (sum over c_in)")
print("and it pays kh*kw*cin*cout weights to do both together.")
print("separating them:")
print(" DEPTHWISE -- one kxk kernel per input channel only.")
print(" POINTWISE -- a 1x1 convolution, all channel mixing.")
print()
print("THE SAVING. a 3x3 convolution, various channel counts:")
print("%12s %12s %16s %16s %12s" % ("in", "out", "standard", "dw+pw", "ratio"))
for cin, cout in ((3, 32), (32, 64), (64, 128), (256, 256), (512, 512)):
s = standard(3, cin, cout)
d = depthwise(3, cin) + pointwise(cin, cout)
print("%12d %12d %16s %16s %12.2fx"
% (cin, cout, "{:,}".format(s), "{:,}".format(d), s / d))
print()
print(" the ratio converges on 1/9 + 1/cout for a 3x3:")
for cout in (32, 128, 512, 4096):
r = 1 / 9 + 1 / cout
print(" cout=%5d -> %.4f (limit %.4f)" % (cout, r, 1 / 9))
print(" so a 3x3 depthwise-separable costs roughly one ninth of normal.")
Output
The next block runs both a standard and a depthwise-separable convolution on the same input, comparing output shapes and weight counts.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
H, W, CIN, COUT = 5, 5, 4, 3
x = rng.normal(0, 1.0, (H, W, CIN))
def conv_standard(x, k):
p = np.pad(x, ((1, 1), (1, 1), (0, 0)), mode="constant")
out = np.zeros((x.shape[0], x.shape[1], k.shape[3]))
for i in range(x.shape[0]):
for j in range(x.shape[1]):
win = p[i:i + 3, j:j + 3, :]
for o in range(k.shape[3]):
out[i, j, o] = (win * k[:, :, :, o]).sum()
return out
def conv_depthwise(x, kd):
p = np.pad(x, ((1, 1), (1, 1), (0, 0)), mode="constant")
out = np.zeros_like(x)
for c in range(x.shape[2]):
for i in range(x.shape[0]):
for j in range(x.shape[1]):
out[i, j, c] = (p[i:i + 3, j:j + 3, c] * kd[:, :, c]).sum()
return out
k_std = rng.normal(0, 0.3, (3, 3, CIN, COUT))
k_dw = rng.normal(0, 0.3, (3, 3, CIN))
k_pw = rng.normal(0, 0.3, (CIN, COUT))
out_std = conv_standard(x, k_std)
out_sep = conv_depthwise(x, k_dw) @ k_pw
print("RUN BOTH on a %dx%dx%d input, %d output channels:" % (H, W, CIN, COUT))
print(" standard -> %s, %d weights" % (out_std.shape, k_std.size))
print(" depthwise + 1x1 -> %s, %d + %d = %d weights"
% (out_sep.shape, k_dw.size, k_pw.size, k_dw.size + k_pw.size))
print(" same output shape, %.1fx fewer weights."
% (k_std.size / (k_dw.size + k_pw.size)))
Output
The next block reveals the capacity each version gets: the standard convolution has full-rank spatial kernels per channel pair, while the separable version is always rank 1.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
CIN, COUT = 4, 3
_ = rng.normal(0, 1.0, (5, 5, CIN)) # advance rng to match
k_std = rng.normal(0, 0.3, (3, 3, CIN, COUT))
k_dw = rng.normal(0, 0.3, (3, 3, CIN))
k_pw = rng.normal(0, 0.3, (CIN, COUT))
def depthwise(kh, cin): return kh * kh * cin
def pointwise(cin, cout): return cin * cout
def standard(kh, cin, cout): return kh * kh * cin * cout
print("WHAT IT GIVES UP:")
print(" standard 3x3, %d->%d: %d distinct spatial kernels"
% (CIN, COUT, CIN * COUT))
print(" depthwise 3x3, %d channels: %d spatial kernels" % (CIN, CIN))
print("%22s %14s %14s" % ("input channel", "standard rank", "separable rank"))
for c in range(CIN):
m_std = k_std[:, :, c, :].reshape(9, COUT)
m_sep = k_dw[:, :, c].reshape(9, 1) * k_pw[c][None, :]
print("%22d %14d %14d"
% (c, np.linalg.matrix_rank(m_std), np.linalg.matrix_rank(m_sep)))
print(" separable is always rank 1: same spatial pattern for every output.")
print()
print("SO THE TRADE IS CAPACITY FOR COST. the usual repair is to go WIDER:")
for width in (1.0, 1.5, 2.0):
cin, cout = int(256 * width), int(256 * width)
d = depthwise(3, cin) + pointwise(cin, cout)
print(" width x%.1f: %s params vs standard 256->256's %s"
% (width, "{:,}".format(d), "{:,}".format(standard(3, 256, 256))))
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does the depthwise step mix?
One k x k filter per input channel, producing one output channel each. The pointwise 1x1 that follows does the channel mixing.
Why does the saving approach k squared for wide layers?
With many output channels the pointwise cost becomes small relative to the full convolution, leaving the k^2 factor - nine for a 3x3, 49 for a 7x7.
Why is the wall-clock speed-up usually smaller than the parameter saving?
Hardware tuned for dense matrix multiplication cannot reach peak throughput on an operation that is starved of arithmetic per memory access.
Cheat sheet
Depthwise Separable Convolution
Split a standard convolution into depthwise and pointwise steps, count the multiply-adds saved, and see why mobile models rely on them.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.