Doing them one at a time
Depthwise. Run one k×k filter per input channel, and keep the channels apart. Channel 7 is convolved with filter 7 and produces channel 7 of the output. Neighbours are mixed; channels are not. Cost: k * k * Cin, so 9 * 64 = 576.
Pointwise. Follow it with a [1×1 convolution](one_by_one_convolutions.html), which combines all the channels at each position and has no spatial extent at all. Channels are mixed; neighbours are not. Cost: Cin * Cout, so 64 * 128 = 8,192.
Together: 8,768 against 73,728 for the same input and output shape. Drag the controls and the ratio in the readout stays close to 8–9× wherever you put them.
Where the ratio comes from
The saving is
k^2 * Cin * Cout
--------------------- = 1 / ( 1/Cout + 1/k^2 )
k^2 * Cin + Cin * Cout
When Cout is large the 1/Cout term nearly vanishes and the ratio approaches k^2 — nine for a 3×3. Set the kernel control to 5 or 7 and the saving rises toward 25 and 49, which is why separable convolutions matter more the larger the kernel.
Set output channels to 8 while leaving input at 64 and the ratio falls, because with few output channels the pointwise step is no longer cheap relative to the whole. The technique earns its keep in wide layers, which is where the cost was.
The cost of the saving
It is not free, and pretending otherwise is the usual mistake.
A full convolution can learn any function of a 3×3×Cin neighbourhood. The separable pair cannot: it is restricted to functions that factor into a spatial part and a channel part. That is a strictly smaller family, and on the same architecture a separable model usually reaches slightly lower accuracy per layer.
What makes it worth doing is that the saving is much larger than the loss. With eight times fewer parameters you can afford more layers, wider layers, or a model that fits on the device at all — and the resulting network usually beats the full-convolution one at equal cost.
Where it is used
MobileNet is built almost entirely from these pairs, and was the paper that made the technique standard for on-device vision.
Xception applied the same argument to Inception, arguing that Inception modules were already approximating a separable convolution and that the extreme version worked better.
EfficientNet uses inverted residual blocks that expand channels with a 1×1, do the spatial work depthwise, and project back down — the same decomposition with an expansion around it.
A note on speed
Parameter count and wall-clock time are not the same thing. A depthwise convolution does very little arithmetic per byte of memory it touches, so it is memory-bound, and hardware optimised for dense matrix multiplication does not reach anything like its peak throughput on one.
An eight-times parameter reduction is therefore often two or three times faster in practice rather than eight. Still worth having, and worth measuring rather than assuming.
Where it goes wrong
Expecting the speed-up to match the parameter count. Measure it.
Using it in the first layer. With three input channels there is almost nothing to save, and the restriction costs accuracy where the model can least afford it.
Forgetting the non-linearity. The depthwise and pointwise steps need an activation between them, or the pair is closer to a single linear map than the two-stage decomposition it is supposed to be.