Gram Matrix for Style

hardPRONeural Style TransferStyle TransferCV: Computational Photography

Implement a function to compute the Gram matrix for neural style transfer, which is a crucial component in capturing the style of an image. This task involves understanding how to represent the style of an image using feature maps extracted from a Convolutional Neural Network (CNN).

The Gram matrix is used to capture the style by computing correlations between feature maps, where the input features is a 3D array of shape $(C, H, W)$ , the output of a CNN layer with $C$ channels, height $H$ , and width $W$ . Each channel's $H \times W$ grid is flattened into a vector, giving the feature map $F$ of shape $(C, H \times W)$ . The Gram matrix $G$ is a 2D tensor that represents the correlations between these feature maps.

Here are the steps to compute the Gram matrix:

Extract the feature map $F$ from a CNN layer.
Compute the correlations between the feature maps. The Gram matrix $G$ can be computed using the formula $G_{ij} = \sum_k F_{ik} F_{jk}$ , which represents the dot product of the $i^{th}$ and $j^{th}$ feature maps. The result is then normalized by the total number of elements $C \cdot H \cdot W$ :

G_{ij} = \frac{1}{C \cdot H \cdot W} \sum_k F_{ik} F_{jk}

Return $G$ as a $(C, C)$ array.

The style loss $L_{style}$ is then computed as the squared difference between the Gram matrix of the content image and the Gram matrix of the style image: $L_{style} = \|G_{content} - G_{style}\|^2$ .

This technique is widely used in image generation and editing tasks.

Example

Input

features = CNN activations shape (64, 32, 32)

Output

Gram matrix shape (64, 64)

Reasoning

features holds $C = 64$ channels, each a $32 \times 32$ grid of activations. Read each channel's grid as one vector of $H \cdot W = 1024$ numbers; channel $i$ becomes row $F_i$ , so $F$ is $64 \times 1024$ .
Entry $G_{ij}$ is the dot product of channel $i$ 's vector with channel $j$ 's: $\sum_{k=1}^{1024} F_{ik} F_{jk}$ . It is large when the two channels fire at the same spatial positions.
Every pair $(i, j)$ of the 64 channels gets one entry, so the result is $64 \times 64$ , matching the output Gram matrix shape (64, 64). The spatial size disappears entirely.
Divide by $C \cdot H \cdot W = 64 \cdot 1024 = 65536$ . Check with all-ones activations: each raw dot product is $1024$ , and $1024 / 65536 = 0.015625 = 1/64$ .
Notice $G$ is symmetric ( $G_{ij} = G_{ji}$ ) and its diagonal $G_{ii}$ is channel $i$ 's energy, never negative. Shuffling pixel positions identically in all channels leaves $G$ unchanged, which is why it captures style rather than layout.

Constraints

features: Feature maps from CNN (C, H, W)
Return: Gram matrix (C, C)

Learn this in

Computational PhotographyCV Study Plan

More from CV: Computational Photography

Trimap-Based Alpha Estimationhard Drago Adaptive Log Mappinghard Alpha Matte Compositingmedium Reinhard Global Tone Mappingmedium Exposure Fusionhard

Concept primerPREVIEW

Background Knowledge

Neural style transfer (NST) separates content and style using a pretrained CNN (e.g., VGG).

Content is captured by feature activations at higher layers (spatial structure, objects).
Style is captured by statistics of features (textures, colors, brush strokes), not their exact spatial locations.

Given a feature map $F \in \mathbb{R}^{C \times (H \times W)}$ , each row is a channel and each column is a spatial location. The Gram matrix $G \in \mathbb{R}^{C \times C}$ is defined as

G_{ij} = \sum_k F_{ik} F_{jk}

It computes inner products between feature channels, measuring how often channels activate together across the image. This is invariant to spatial arrangement (order of columns $k$ ), so it captures texture / style rather than layout. Style loss then compares Gram matrices of style vs generated image:

L_{style} = \|G^{(gen)} - G^{(style)}\|_F^2

Algorithm / Approach

General pattern to compute a Gram matrix for style:

Take a CNN feature map of shape $(C, H, W)$ .
Flatten spatial dimensions to get shape $(C, H \times W)$ .
Compute Gram matrix as a channel–channel correlation: $G = F \cdot F^\top$ , giving shape $(C, C)$ .
Optionally normalize (e.g., divide by $H \times W$ or $C \cdot H \cdot W$ ) so style loss scales reasonably across layers.

In NST, this Gram computation is applied to multiple style layers, and their Gram matrices are compared to the style image’s Gram matrices to form the total style loss.

Step-by-Step Strategy

Assume input feature map tensor F has shape (C, H, W) (or (N, C, H, W) with batch):

Handle batch (if present)

If F is (N, C, H, W), usually you:
Either process a single image (N=1) and drop batch dimension.
Or compute Gram per sample independently.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

Example

Constraints

Learn this in

More from CV: Computational Photography

Continue the full explanation

Editor locked