Depthwise Separable Convolution

Depthwise separable convolution replaces one expensive convolution with two cheaper ones: a depthwise spatial convolution applied independently per input channel, followed by a pointwise $1\times1$ convolution that mixes channel information.$This preserves the same overall mapping from C_{in} \times H \times W to C_{out} \times H' \times W', but with far fewer parameters and FLOPs, which is why it is central to MobileNet-style architectures.$

---

1. Background Knowledge (key concepts)

• Standard 2D convolution in CNNs

A standard convolution layer with kernel size K \times K maps an input of shape (C_{in}, H, W) to an output of shape (C_{out}, H', W') using a weight tensor of shape (C_{out}, C_{in}, K, K). Each output channel is a weighted sum over all input channels and spatial neighbors, so both spatial and cross-channel mixing happen in a single operation.

• Depthwise separable convolution

MobileNet factorizes this into two steps:

1. Depthwise convolution

• One K \times K filter per input channel.
• Weights: shape (C_{in}, 1, K, K), often stored as (C_{in}, K, K).
• Each channel is convolved independently (no mixing between channels here).

2. Pointwise convolution

• A 1 \times 1 convolution across channels to mix them.
• Weights: shape (C_{out}, C_{in}, 1, 1), typically viewed as a matrix (C_{out}, C_{in}).

This dramatically reduces computational cost while maintaining accuracy for many vision tasks.

• Parameter / FLOP reduction

• Standard conv params:

$$P_{\text{std}} = C_{out} \cdot C_{in} \cdot K^2$$

• Depthwise separable params:

$$P_{\text{dwsep}} = C_{in} \cdot K^2 + C_{out} \cdot C_{in}$$

• Relative cost:

$$\frac{P_{\text{dwsep}}}{P_{\text{std}}} = \frac{1}{C_{out}} + \frac{1}{K^2}$$

The same ratio holds (up to constants) for multiply-add operations, which explains the efficiency gain often cited for MobileNets.

---

2. Algorithm / Approach Pattern

Conceptually, the algorithm is:

1. Depthwise pass For each input channel c:

• Convolve the 2D slice X_c \in \mathbb{R}^{H \times W} with its own K \times K kernel W^{dw}_c.
• Store the result as an intermediate “depthwise output” of shape (C_{in}, H', W').

2. Pointwise pass For each output channel o:

• Take the vector at each spatial position (i,j) across all depthwise channels (C_{in}).
• Multiply by a 1 \times 1 kernel (a vector) W^{pw}o \in \mathbb{R}^{C{in}}.
• This is equivalent to a matrix multiplication between a (C_{out} \times C_{in}) weight matrix and the C_{in}-dim feature vector at each spatial position.

In code terms (framework-agnostic): first perform a grouped convolution with groups = C_in, then a standard 1\times1 convolution.

---

3. Step-by-Step Strategy to Implement

Assume input tensor shape: (N, C_in, H, W).

1. Understand required shapes

• Depthwise weights: (C_in, 1, K, K) or equivalently (C_in, K, K) depending on the API.
• Pointwise weights: (C_out, C_in, 1, 1) or equivalently (C_out, C_in).

2. Implement depthwise convolution

• Loop over batch and input channels.
• For each channel, apply a 2D convolution with:
• Stride, padding, and dilation matching the original full conv.
• No mixing between channels.
• Store result in an intermediate tensor dw_out of shape (N, C_in, H', W').