Gradient Flow Analysis

hardPROGradient ManipulationPyTorch Advanced

Problem Statement

Analyze gradient flow through a deep network to detect vanishing or exploding gradients.

Background

In deep networks, gradients can vanish (approach 0) or explode (grow huge) as they flow backward through layers. Monitoring gradient norms per layer helps diagnose training issues.

Your Task

The starter code creates a 5-layer Sigmoid network, runs a forward pass, and calls backward. Compute the L2 gradient norm for each layer's weights to analyze how gradients flow through the network.

The rest (vanishing detection, min/max) is handled by the pre-filled code.

Output Format

Returns a dictionary with "grad_norms" (list of 5), "vanishing", "max_norm", and "min_norm".

Example

Input

None

Output

{'grad_norms': [0.000149, 0.000411, 0.001311, 0.004769, 0.059498], 'vanishing': True, 'max_norm': 0.059498, 'min_norm': 0.000149}

Reasoning

The function gradient_flow_test() starts by seeding the random number generator with torch.manual_seed(42) to ensure reproducibility.
A 5-layer network is created with nn.Linear(4, 4) layers and Sigmoid activations, then the input torch.ones(1, 4) and target torch.zeros(1, 4) are used to compute the MSE loss.
The backward() function is called to compute the gradients, and for each Linear layer, the L2 norm of the weight gradients is calculated as $\sqrt{\sum_{i=1}^{n} g_i^2}$ , where $g_i$ is the $i^{th}$ gradient element, resulting in the list of gradient norms: [0.000149, 0.000411, 0.001311, 0.004769, 0.059498].
The function then determines if the gradients are vanishing by checking if the last layer's gradient norm is less than 0.01 times the first layer's gradient norm, resulting in vanishing being True, and calculates the maximum and minimum gradient norms across layers.

Constraints

5 Linear(4,4) layers with Sigmoid activation
Monitor weight gradient norms per layer
Detect vanishing gradients

Learn this in

Deep LearningCV Study Plan PyTorch FundamentalsFoundations Sequence ModelsNLP Study Plan

More from PyTorch Advanced

Implement SGD with Momentumhard Implement Custom SGD Optimizerhard Weight Decay for Specific Layersmedium Freeze Backbone, Train Headmedium Per-Layer Learning Rateseasy

Concept primerPREVIEW

Background Knowledge

The problem of gradient flow analysis is crucial in understanding the behavior of deep neural networks during training. In deep learning, gradients are used to update the model's parameters to minimize the loss function. However, as gradients flow backward through the layers, they can suffer from two main issues: vanishing gradients and exploding gradients. Vanishing gradients occur when the gradients become very small, making it difficult for the model to learn. On the other hand, exploding gradients happen when the gradients become very large, causing the model's parameters to update excessively.

The chain rule of calculus is used to compute the gradients in a neural network. When applying the chain rule, the gradients are multiplied together, which can lead to vanishing or exploding gradients. To diagnose these issues, monitoring the gradient norms per layer is helpful. The gradient norm is a measure of the magnitude of the gradient, and it can be computed using the L2 norm (also known as the Euclidean norm). By analyzing the gradient norms, developers can identify potential problems in the network and take corrective actions, such as adjusting the learning rate or using gradient clipping.

In the context of this problem, the mean squared error (MSE) loss function is used, which is a common choice for regression tasks. The backward pass is used to compute the gradients of the loss function with respect to the model's parameters. The Sigmoid activation function is used between the linear layers, which can contribute to vanishing gradients due to its saturating nature. Understanding these concepts is essential to solving the problem and analyzing the gradient flow in the given network.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Create a deep neural network with the specified architecture
Define the loss function and compute the gradients using the backward pass
Analyze the gradient norms per layer to diagnose potential issues
Compute the required metrics, such as the gradient norms, vanishing gradient indicator, max norm, and min norm

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results