Selective Parameter Freezing

mediumPROGradient ManipulationPyTorch Advanced

Problem Statement

Freeze specific layers of a model while keeping others trainable.

Background

In transfer learning, you often freeze pretrained layers and only train the top layers. You can freeze parameters by setting requires_grad = False.

Your Task

The starter code creates a 2-layer model. Freeze layer1's parameters so they don't update during training, then count the total, trainable, and frozen parameters in the model.

Output Format

Returns a dictionary with "total_params", "trainable_params", "frozen_params", and "layer1_frozen".

Example

Input

None

Output

{'total_params': 58, 'trainable_params': 18, 'frozen_params': 40, 'layer1_frozen': True}

Reasoning

The model is created with nn.Sequential containing two linear layers: nn.Linear(4, 8) (layer1) and nn.Linear(8, 2) (layer2).
Layer1 has $4 \cdot 8 + 8 = 40$ parameters (weights and bias), and layer2 has $8 \cdot 2 + 2 = 18$ parameters, making a total of $40 + 18 = 58$ parameters.
By setting requires_grad=False on layer1's parameters, all 40 parameters in layer1 are frozen, leaving 18 parameters in layer2 trainable.
The function then returns a dictionary with the total parameter count (58), trainable parameter count (18), frozen parameter count (40), and a boolean indicating that all layer1 parameters are frozen (True).

Constraints

Use nn.Sequential with named layers
Freeze by setting requires_grad = False
Count parameters using .numel()

Learn this in

PyTorch FundamentalsFoundations

More from PyTorch Advanced

Implement SGD with Momentumhard Implement Custom SGD Optimizerhard Weight Decay for Specific Layersmedium Freeze Backbone, Train Headmedium Per-Layer Learning Rateseasy

Concept primerPREVIEW

Background Knowledge

Introduction to Transfer Learning

Transfer learning is a technique in machine learning where a model trained on one task is re-purposed or fine-tuned for another related task. This approach is particularly useful when there is limited training data for the new task, as the pre-trained model has already learned general features that can be applied to the new task.

Understanding Model Layers and Parameters

In the context of neural networks, a model consists of multiple layers, each with its own set of parameters (weights and biases). These parameters are learned during the training process and are used to make predictions on new, unseen data. In transfer learning, it's common to freeze the parameters of the earlier layers (which have learned general features) and only train the parameters of the later layers (which learn task-specific features).

Parameter Freezing and Gradient Manipulation

Parameter freezing involves setting the requires_grad attribute of a parameter to False, which prevents the parameter from being updated during backpropagation. This is useful in transfer learning, as it allows us to preserve the pre-learned features while still training the model on the new task. Gradient manipulation refers to the process of controlling how gradients are computed and applied to model parameters during training. This can include techniques such as gradient clipping, gradient normalization, and parameter freezing.

Algorithm/Approach

The general approach to solving this problem involves creating a PyTorch model with the specified layers, freezing the parameters of the desired layer, and then counting the total, trainable, and frozen parameters. This can be achieved by iterating over the model's parameters and checking their requires_grad attribute.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results