Compare Scheduler Curves
Problem Statement
Compare three different schedulers side by side over the same number of epochs.
Background
Different schedulers produce different decay curves. StepLR gives discrete drops, ExponentialLR gives smooth decay, CosineAnnealingLR gives a cosine-shaped curve.
Your Task
The starter code provides a run_scheduler helper function. Use it to generate LR curves for StepLR (step_size=5, gamma=0.5), ExponentialLR (gamma=0.9), and CosineAnnealingLR (T_max=15, eta_min=0.001).
Output Format
Returns a dictionary with "step_lrs", "exp_lrs", "cosine_lrs", and "final_comparison".
Example:
None
{'step_lrs': [0.1, 0.1, 0.1, 0.1, 0.1, 0.05, 0.05, 0.05, 0.05, 0.05, 0.025, 0.025, 0.025, 0.025, 0.025], 'exp_lrs': [0.1, 0.09, 0.081, 0.0729, 0.0656, 0.059, 0.0531, 0.0478, 0.043, 0.0387, 0.0349, 0.0314, 0.0282, 0.0254, 0.0229], 'cosine_lrs': [0.1, 0.0966, 0.0868, 0.0714, 0.0517, 0.0505, 0.0302, 0.0302, 0.0143, 0.0143, 0.001, 0.001, 0.001, 0.001, 0.001], 'final_comparison': {'step': 0.025, 'exponential': 0.0229, 'cosine': 0.001}}- We initialize three schedulers: StepLR with lr=0.1, step_size=5, and γ=0.5; ExponentialLR with lr=0.1 and γ=0.9; and CosineAnnealingLR with lr=0.1, T_max=15, and η_min=0.001.
- For each of the 15 epochs, we update the learning rate for each scheduler: StepLR reduces the rate by a factor of γ every step_size epochs, ExponentialLR reduces the rate by a factor of γ every epoch, and CosineAnnealingLR reduces the rate according to a cosine schedule.
- We record the learning rate values for each scheduler at each epoch, rounding to 4 decimals, resulting in the
'step_lrs','exp_lrs', and'cosine_lrs'lists in the output. - Finally, we create the
'final_comparison'dictionary with the final learning rate of each scheduler, giving the output{'step': 0.025, 'exponential': 0.0229, 'cosine': 0.001}.
Constraints:
- Same initial LR for all three
- 15 epochs each
- Compare final values
Background Knowledge
Introduction to Learning Rate Schedulers
Learning Rate Schedulers are a crucial component in training neural networks. They control the learning rate, which determines how quickly the model learns from the training data. A high learning rate can lead to fast convergence but may also cause the model to overshoot the optimal solution, while a low learning rate can result in slow convergence. There are several types of learning rate schedulers, including StepLR, ExponentialLR, and CosineAnnealingLR, each with its own strengths and weaknesses.
Understanding Different Schedulers
- StepLR: This scheduler reduces the learning rate by a factor of gamma every step_size epochs. It's a simple and effective way to adjust the learning rate during training.
- ExponentialLR: This scheduler reduces the learning rate exponentially over time, with the learning rate at each epoch being gamma times the learning rate at the previous epoch.
- CosineAnnealingLR: This scheduler reduces the learning rate using a cosine annealing schedule, which starts with a high learning rate and gradually decreases it to a minimum value over a specified number of epochs.
Importance of Comparing Schedulers
Comparing different learning rate schedulers is essential to determine which one works best for a specific problem. Each scheduler has its own hyperparameters that need to be tuned, and the choice of scheduler can significantly impact the performance of the model. By comparing the learning rate curves of different schedulers, we can gain insights into how they affect the training process and choose the best one for our specific use case.
Algorithm/Approach
The general approach to solving this problem involves creating a neural network model with a simple architecture (e.g., nn.Linear(2, 1)) and defining three different learning rate schedulers. We then simulate the training process for a specified number of epochs, recording the learning rate at each epoch for each scheduler. Finally, we compare the learning rate curves of the three schedulers and calculate the final learning rate for each one.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.