Compare Scheduler Curves

mediumPROLearning Rate SchedulersPyTorch Advanced

Problem Statement

Compare three different schedulers side by side over the same number of epochs.

Background

Different schedulers produce different decay curves. StepLR gives discrete drops, ExponentialLR gives smooth decay, CosineAnnealingLR gives a cosine-shaped curve.

Your Task

The starter code provides a run_scheduler helper function. Use it to generate LR curves for StepLR (step_size=5, gamma=0.5), ExponentialLR (gamma=0.9), and CosineAnnealingLR (T_max=15, eta_min=0.001).

Output Format

Returns a dictionary with "step_lrs", "exp_lrs", "cosine_lrs", and "final_comparison".

Example

Input

None

Output

{'step_lrs': [0.1, 0.1, 0.1, 0.1, 0.1, 0.05, 0.05, 0.05, 0.05, 0.05, 0.025, 0.025, 0.025, 0.025, 0.025], 'exp_lrs': [0.1, 0.09, 0.081, 0.0729, 0.0656, 0.059, 0.0531, 0.0478, 0.043, 0.0387, 0.0349, 0.0314, 0.0282, 0.0254, 0.0229], 'cosine_lrs': [0.1, 0.0966, 0.0868, 0.0714, 0.0517, 0.0505, 0.0302, 0.0302, 0.0143, 0.0143, 0.001, 0.001, 0.001, 0.001, 0.001], 'final_comparison': {'step': 0.025, 'exponential': 0.0229, 'cosine': 0.001}}

Reasoning

We initialize three schedulers: StepLR with $lr=0.1$ , $step\_size=5$ , and $\gamma=0.5$ ; ExponentialLR with $lr=0.1$ and $\gamma=0.9$ ; and CosineAnnealingLR with $lr=0.1$ , $T\_max=15$ , and $\eta\_min=0.001$ .
For each of the 15 epochs, we update the learning rate for each scheduler: StepLR reduces the rate by a factor of $\gamma$ every $step\_size$ epochs, ExponentialLR reduces the rate by a factor of $\gamma$ every epoch, and CosineAnnealingLR reduces the rate according to a cosine schedule.
We record the learning rate values for each scheduler at each epoch, rounding to 4 decimals, resulting in the 'step_lrs', 'exp_lrs', and 'cosine_lrs' lists in the output.
Finally, we create the 'final_comparison' dictionary with the final learning rate of each scheduler, giving the output {'step': 0.025, 'exponential': 0.0229, 'cosine': 0.001}.

Constraints

Same initial LR for all three
15 epochs each
Compare final values

More from PyTorch Advanced

Implement SGD with Momentumhard Implement Custom SGD Optimizerhard Weight Decay for Specific Layersmedium Freeze Backbone, Train Headmedium Per-Layer Learning Rateseasy

Concept primerPREVIEW

Background Knowledge

Introduction to Learning Rate Schedulers

Learning Rate Schedulers are a crucial component in training neural networks. They control the learning rate, which determines how quickly the model learns from the training data. A high learning rate can lead to fast convergence but may also cause the model to overshoot the optimal solution, while a low learning rate can result in slow convergence. There are several types of learning rate schedulers, including StepLR, ExponentialLR, and CosineAnnealingLR, each with its own strengths and weaknesses.

Understanding Different Schedulers

StepLR: This scheduler reduces the learning rate by a factor of gamma every step_size epochs. It's a simple and effective way to adjust the learning rate during training.
ExponentialLR: This scheduler reduces the learning rate exponentially over time, with the learning rate at each epoch being gamma times the learning rate at the previous epoch.
CosineAnnealingLR: This scheduler reduces the learning rate using a cosine annealing schedule, which starts with a high learning rate and gradually decreases it to a minimum value over a specified number of epochs.

Importance of Comparing Schedulers

Comparing different learning rate schedulers is essential to determine which one works best for a specific problem. Each scheduler has its own hyperparameters that need to be tuned, and the choice of scheduler can significantly impact the performance of the model. By comparing the learning rate curves of different schedulers, we can gain insights into how they affect the training process and choose the best one for our specific use case.

Algorithm/Approach

The general approach to solving this problem involves creating a neural network model with a simple architecture (e.g., nn.Linear(2, 1)) and defining three different learning rate schedulers. We then simulate the training process for a specified number of epochs, recording the learning rate at each epoch for each scheduler. Finally, we compare the learning rate curves of the three schedulers and calculate the final learning rate for each one.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results