Freeze Backbone, Train Head
Problem Statement
Implement the standard transfer learning pattern: freeze the backbone and only optimize the head.
Background
In transfer learning, you freeze pretrained layers (backbone) and only train new layers (head). Frozen parameters don't receive gradient updates.
Your Task
The starter code creates a backbone and head. Freeze the backbone parameters so they won't be updated, and create an optimizer that only trains the head.
The training loop and verification are pre-filled.
Output Format
Returns a dictionary with "backbone_changed" (should be False), "head_changed" (should be True), "trainable_params", and "total_params".
Example
Input
None
Output
{'backbone_changed': False, 'head_changed': True, 'trainable_params': 10, 'total_params': 78}Reasoning
- The function
freeze_backbone_test()starts by seeding withtorch.manual_seed(42)to ensure reproducibility, then creates a backbone with two linear layers and ReLU activations, and a head with one linear layer. - The backbone parameters are frozen, meaning their values will not be updated during training, while the head parameters are left trainable.
- The function then runs 3 training steps with input
[[1.0, 2.0, 3.0, 4.0]]and target[[1.0, 0.0]], which updates the head parameters but leaves the backbone parameters unchanged. - After training, the function checks for changes in the backbone and head parameters, counts the number of trainable and total parameters, and returns a dictionary with the results, including
trainable_paramsbeing the number of parameters in the head (10 in this case, since a linear layer with input size 4 and output size 2 has parameters) andtotal_paramsbeing the total number of parameters in the model (78 in this case).
Constraints
- Freeze backbone with requires_grad=False
- Only pass head params to optimizer
- Verify backbone unchanged after training