Multi-Scale Feature Pyramid: Background & Implementation Guide

Background Knowledge

A feature pyramid is a hierarchical representation of an image at multiple scales, where each level captures information at different resolutions. The key insight is that real-world objects vary dramatically in size—a person might be 500 pixels tall in one image and 50 pixels in another. By creating a pyramid of progressively downsampled versions of the image, detection systems can identify objects regardless of their scale. This multi-scale approach is fundamental to modern computer vision, used in classical methods like SIFT and HOG, as well as deep learning architectures like Feature Pyramid Networks (FPN).

The Gaussian blur before downsampling step is critical for preventing aliasing—a phenomenon where high-frequency information (fine details) gets incorrectly represented at lower resolutions, violating the Nyquist sampling criterion. Without blur, downsampling can introduce artifacts and loss of information. The blur acts as a low-pass filter, removing frequencies that cannot be represented at the lower resolution, ensuring smooth transitions between pyramid levels.

The pyramid structure enables coarse-to-fine processing: upper levels (smaller images) capture large-scale structures and global context, while lower levels (larger images) preserve fine details and local information. This hierarchical representation is computationally efficient because processing is done at multiple resolutions rather than only at full resolution, and it provides rich feature representations for downstream tasks like object detection and segmentation.

Algorithm/Approach

The general approach follows a sequential downsampling pipeline:

1. Initialize with the original image as level 0
2. Iterate to create subsequent levels by:

• Applying Gaussian blur to the current level
• Downsampling (reducing spatial dimensions by factor of 2)
• Storing the result as the next pyramid level

3. Terminate when reaching a minimum size threshold (typically when either dimension becomes too small to be useful)

This creates a bottom-up pyramid where each level is progressively smaller and more abstract. The key is maintaining proper blur parameters and downsampling factors to ensure smooth, artifact-free transitions between scales.

Step-by-Step Strategy

Step 1: Set up your pyramid structure

• Decide on the number of levels (4-6 is typical, but depends on input image size)
• Create a container (list or dictionary) to store pyramid levels
• Define blur kernel size and downsampling factor (typically 2)

Step 2: Implement Gaussian blur

• Use PyTorch's torch.nn.functional.gaussian_blur() or torchvision.transforms.GaussianBlur
• Choose appropriate kernel size (typically 5×5 or 7×7) and sigma value
• Apply blur to the current level before downsampling

Step 3: Implement downsampling

• Use torch.nn.functional.interpolate() with mode='bilinear' or 'nearest'
• Reduce spatial dimensions by factor of 2 (e.g., 512×512 → 256×256)
• Alternatively, use torch.nn.MaxPool2d or torch.nn.AvgPool2d

Step 4: Build the pyramid loop

• Start with the original image
• For each subsequent level, apply blur then downsample
• Store each level in your container
• Continue until reaching a stopping condition (minimum size or fixed number of levels)

Step 5: Validate your pyramid

• Check that each level is exactly half the size of the previous level
• Verify that blur is actually being applied (compare blurred vs. unblurred)
• Ensure no unexpected shape changes or dimension mismatches

Common Pitfalls

• Forgetting the blur step: Downsampling without blur causes aliasing artifacts and information loss. Always blur first.
• Incorrect blur parameters: Using too small a kernel or sigma won't effectively remove high frequencies. Use at least 5×5 kernels with sigma ≈ 1.0-2.0.
• Dimension mismatches: When downsampling, ensure your output dimensions are exactly half. Watch for off-by-one errors with odd-sized images.
• Channel handling: If working with multi-channel images (RGB), ensure blur and downsampling operations preserve all channels correctly.
• Stopping condition: Don't create levels that are too small (e.g., 1×1 pixels). Set a reasonable minimum size threshold.
• Data type issues: Ensure consistent tensor types (float32) throughout the pipeline, especially when using interpolation.
• Memory efficiency: Storing all pyramid levels uses significant memory. Consider whether you need to keep all levels or can generate them on-the-fly.

Time & Space Complexity

Time Complexity: $O(n^2 \cdot k)$ where $n$ is the original image dimension and $k$ is the number of pyramid levels.

• Each blur operation is $O(n^2)$ for an $n \times n$ image
• Downsampling is also $O(n^2)$
• Since each level is half the size of the previous, total work is $n^2 + (n/2)^2 + (n/4)^2 +... \approx O(n^2)$ across all levels
• With $k$ levels, this becomes $O(n^2 \cdot k)$, though typically $k = O(\log n)$

Space Complexity: $O(n^2 \cdot k)$ to store all pyramid levels.

• Level 0: $n^2$ pixels
• Level 1: $(n/2)^2$ pixels
• Level $i$: $(n/2^i)^2$ pixels
• Total: $n^2(1 + 1/4 + 1/16 +...) \approx 1.33 \cdot n^2$, which is $O(n^2)$ with a constant factor
• Multiplied by $k$ levels and any channel dimensions

The geometric series means the total space is dominated by the first (largest) level, making the pyramid surprisingly memory-efficient compared to storing multiple full-resolution copies.