📘

Gradient Clipping by Norm

Problem Statement

Implement gradient clipping by norm to prevent exploding gradients.

Background

Norm-based gradient clipping scales all gradients so their total L2 norm doesn't exceed a threshold. This is essential for training RNNs and Transformers.

Your Task

The starter code creates a model, computes a loss, and calls backward. Compute the total gradient norm before clipping, apply norm-based gradient clipping with max_norm=1.0, then compute the norm again after clipping.

Output Format

Returns a dictionary with "norm_before", "norm_after", and "was_clipped".

Example:

Input:

None

Output:

{'norm_before': 892.5765, 'norm_after': 1.0, 'was_clipped': True}

Reasoning:

We create a simple model nn.Linear(4, 2) and seed with torch.manual_seed(42) to ensure reproducibility.
The input [[10.0, 20.0, 30.0, 40.0]] and target [[1.0, 1.0]] are used to compute the MSE loss, which is then used to calculate the gradients using the backward method.
The total gradient norm BEFORE clipping is calculated and recorded, resulting in a value of $892.5765$ , which exceeds the max_norm threshold of $1.0$ .
The gradients are then clipped using torch.nn.utils.clip_grad_norm_ with max_norm=1.0, resulting in a total gradient norm AFTER clipping of $1.0$ , and since the norm before clipping was greater than $1.0$ , was_clipped is set to True.

Constraints:

Use torch.nn.utils.clip_grad_norm_
max_norm=1.0
Use MSE loss

Editor

Python 3.13.1

Test Results

0/0

Run code to see test results.

📘