CosineAnnealingLR Scheduler
Problem Statement
Use CosineAnnealingLR for cosine-shaped learning rate decay.
Background
CosineAnnealingLR follows a cosine curve from initial LR down to eta_min over T_max epochs, then back up.
Your Task
The starter code creates a model and SGD optimizer. Create a CosineAnnealingLR scheduler that decays over 20 epochs to a minimum LR of 0.001.
The epoch loop is pre-filled.
Output Format
Returns a dictionary with "lr_history" (20 values), "max_lr", "min_lr", and "mid_lr".
Example:
None
{'lr_history': [0.1, 0.0976, 0.0905, 0.079, 0.0638, 0.0505, 0.0362, 0.021, 0.0095, 0.0034, 0.001, 0.0034, 0.0095, 0.021, 0.0362, 0.0505, 0.0638, 0.079, 0.0905, 0.0976], 'max_lr': 0.1, 'min_lr': 0.001, 'mid_lr': 0.001}- The
cosine_lr_test()function initializes a learning rate of 0.1 and creates aCosineAnnealingLRscheduler with Tmax=20 and ηmin=0.001. - The scheduler applies a cosine-shaped learning rate decay: lr=ηmin+21(η−ηmin)(1+cos(Tmaxπt)), where t is the current epoch and η is the initial learning rate.
- Over 20 epochs, the learning rate decays and then increases, following the cosine curve, resulting in the recorded
lr_historyvalues. - The maximum, minimum, and mid-point learning rates are extracted from the
lr_historyto produce the final output dictionary values:max_lr,min_lr, andmid_lr.
Constraints:
- T_max=20, eta_min=0.001
- Initial lr=0.1
- 20 epochs
Background Knowledge
The CosineAnnealingLR scheduler is a type of learning rate scheduler in PyTorch that uses a cosine-shaped decay schedule. This means that the learning rate will start at a maximum value, decay to a minimum value, and then repeat this process. The CosineAnnealingLR scheduler is often used in conjunction with the SGD (Stochastic Gradient Descent) optimizer. The SGD optimizer is a widely used optimization algorithm that updates the model's parameters based on the gradient of the loss function.
The CosineAnnealingLR scheduler takes several parameters, including T_max (the maximum number of iterations) and eta_min (the minimum learning rate). The learning rate is calculated using the formula: lr=ηmin+21(ηmax−ηmin)(1+cos(Tmaxπt)), where t is the current iteration, ηmax is the maximum learning rate, and ηmin is the minimum learning rate. This formula produces a cosine-shaped decay schedule, where the learning rate starts at ηmax, decays to ηmin, and then repeats.
Understanding the CosineAnnealingLR scheduler requires knowledge of learning rate schedulers and their role in deep learning. Learning rate schedulers are used to adjust the learning rate during training, which can help improve the convergence of the model. The CosineAnnealingLR scheduler is just one example of a learning rate scheduler, and there are many other types of schedulers available in PyTorch, each with their own strengths and weaknesses.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.