Depthwise Separable Convolution

Implement depthwise separable convolution, a key efficiency technique in MobileNets.

Standard convolution: $C_{in} \times H \times W \rightarrow C_{out} \times H' \times W'$ Parameters: $C_{out} \times C_{in} \times K \times K$

Depthwise separable breaks this into (both steps use stride 1 and no padding, so a $K \times K$ kernel shrinks each spatial side to $H - K + 1$ ):

Depthwise: Each input channel convolved separately
- Parameters: $C_{in} \times K \times K$
Pointwise: 1×1 convolution to mix channels
- Parameters: $C_{out} \times C_{in}$

Efficiency gain: $\frac{1}{C_{out}} + \frac{1}{K^2}$ of standard conv!

Example

Input

input: (1, 3, 8, 8)  # RGB image
depthwise: (3, 1, 3, 3)  # 3x3 per channel
pointwise: (16, 3, 1, 1)  # Expand to 16 channels

Output

tensor of shape (1, 16, 6, 6)

Reasoning

Inputs: x is a batch of 1 image with $C_{in}=3$ channels on an $8 \times 8$ grid; the depthwise kernel holds one $3 \times 3$ filter per channel ( $3 \times 1 \times 3 \times 3$ ); the pointwise kernel holds a weight $P_{o,c}$ per (output, input) channel pair ( $16 \times 3 \times 1 \times 1$ ).
Depthwise step: channel $c$ is filtered only by its own kernel, $D_c(i,j) = \sum_{u,v} x_c(i+u, j+v)\,k_c(u,v)$ . With no padding and stride 1, a $3 \times 3$ window fits in $8 - 3 + 1 = 6$ positions per axis: $1 \times 3 \times 6 \times 6$ , channels unmixed.
Pointwise step: at every pixel, $y_o(i,j) = \sum_{c=1}^{3} P_{o,c}\,D_c(i,j)$ , which mixes the 3 channels into 16 without touching the $6 \times 6$ grid: tensor of shape (1, 16, 6, 6).
Check with all-ones tensors: each depthwise value is $9$ , each output value is $3 \times 9 = 27$ .
Notice the savings: $3 \cdot 9 + 16 \cdot 3 = 75$ weights versus $16 \cdot 3 \cdot 9 = 432$ for a standard conv, a ratio of $\frac{1}{16} + \frac{1}{9} \approx 0.174$ .

Constraints

input: Tensor (batch, channels, height, width)
depthwise_kernel: (channels, 1, k, k)
pointwise_kernel: (out_channels, channels, 1, 1)
Return: Output tensor

More from CV: Deep Learning

Implement Max Pooling with Indiceshard Softmax Cross-Entropy Gradientmedium Batch Normalization Forward Passhard Flatten Feature Mapeasy Max Poolingeasy

Concept primerPREVIEW

Depthwise separable convolution replaces one expensive convolution with two cheaper ones: a depthwise spatial convolution applied independently per input channel, followed by a pointwise $1\times1$ convolution that mixes channel information. $This preserves the same overall mapping from C_{in} \times H \times W to C_{out} \times H' \times W', but with far fewer parameters and FLOPs, which is why it is central to MobileNet-style architectures.$

1. Background Knowledge (key concepts)

Standard 2D convolution in CNNs

A standard convolution layer with kernel size K \times K maps an input of shape (C_{in}, H, W) to an output of shape (C_{out}, H', W') using a weight tensor of shape (C_{out}, C_{in}, K, K). Each output channel is a weighted sum over all input channels and spatial neighbors, so both spatial and cross-channel mixing happen in a single operation.

Depthwise separable convolution

MobileNet factorizes this into two steps:

Depthwise convolution

One K \times K filter per input channel.
Weights: shape (C_{in}, 1, K, K), often stored as (C_{in}, K, K).
Each channel is convolved independently (no mixing between channels here).

Pointwise convolution

A 1 \times 1 convolution across channels to mix them.
Weights: shape (C_{out}, C_{in}, 1, 1), typically viewed as a matrix (C_{out}, C_{in}).

This dramatically reduces computational cost while maintaining accuracy for many vision tasks.

Parameter / FLOP reduction
Standard conv params:

P_{\text{std}} = C_{out} \cdot C_{in} \cdot K^2

Depthwise separable params:

P_{\text{dwsep}} = C_{in} \cdot K^2 + C_{out} \cdot C_{in}

Relative cost:

\frac{P_{\text{dwsep}}}{P_{\text{std}}} = \frac{1}{C_{out}} + \frac{1}{K^2}

The same ratio holds (up to constants) for multiply-add operations, which explains the efficiency gain often cited for MobileNets.

2. Algorithm / Approach Pattern

Conceptually, the algorithm is:

Depthwise pass For each input channel c:

Convolve the 2D slice X_c \in \mathbb{R}^{H \times W} with its own K \times K kernel W^{dw}_c.
Store the result as an intermediate “depthwise output” of shape (C_{in}, H', W').

Pointwise pass For each output channel o:

Take the vector at each spatial position (i,j) across all depthwise channels (C_{in}).
Multiply by a 1 \times 1 kernel (a vector) W^{pw}o \in \mathbb{R}^{C{in}}.
This is equivalent to a matrix multiplication between a (C_{out} \times C_{in}) weight matrix and the C_{in}-dim feature vector at each spatial position.

In code terms (framework-agnostic): first perform a grouped convolution with groups = C_in, then a standard 1\times1 convolution.

3. Step-by-Step Strategy to Implement

Assume input tensor shape: (N, C_in, H, W).

Understand required shapes

Depthwise weights: (C_in, 1, K, K) or equivalently (C_in, K, K) depending on the API.
Pointwise weights: (C_out, C_in, 1, 1) or equivalently (C_out, C_in).

Implement depthwise convolution

Loop over batch and input channels.
For each channel, apply a 2D convolution with:
Stride, padding, and dilation matching the original full conv.
No mixing between channels.
Store result in an intermediate tensor dw_out of shape (N, C_in, H', W').

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results