📘

ExponentialLR Scheduler

EasyLearning Rate Schedulers 📚 PyTorch Advanced

Problem Statement

Use ExponentialLR for smooth exponential decay of the learning rate.

Background

ExponentialLR multiplies the learning rate by gamma every epoch: lr[n] = initial_lr * gamma^n.

Your Task

The starter code creates a model and SGD optimizer. Create an ExponentialLR scheduler with gamma=0.9.

The epoch loop and exponential verification are pre-filled.

Output Format

Returns a dictionary with "lr_history" (10 values), "decay_ratio", and "is_exponential".

Example:

Input:

None

Output:

{'lr_history': [1.0, 0.9, 0.81, 0.729, 0.6561, 0.5905, 0.5314, 0.4783, 0.4305, 0.3874], 'decay_ratio': 0.3874, 'is_exponential': True}

Reasoning:

We create an nn.Linear(2, 1) model with SGD optimizer and a learning rate of 1.0.
An ExponentialLR scheduler is applied to the optimizer with a gamma value of 0.9, which means the learning rate will decay by a factor of $0.9$ at each epoch.
Over 10 epochs, the learning rate is updated at each epoch, resulting in a smooth exponential decay: $1.0, 1.0 \cdot 0.9, 1.0 \cdot 0.9^2, ..., 1.0 \cdot 0.9^9$ , yielding the lr_history list.
The decay_ratio is calculated as the ratio of the last learning rate to the first, which is $0.9^9$ , and is_exponential is set to True since each learning rate is approximately $0.9$ times the previous one.

Constraints:

gamma=0.9
Initial lr=1.0
10 epochs

More from PyTorch Advanced

Implement SGD with MomentumHard Implement Custom SGD OptimizerHard Weight Decay for Specific LayersMedium Freeze Backbone, Train HeadMedium Per-Layer Learning RatesEasy

Background Knowledge

The ExponentialLR Scheduler is a type of learning rate scheduler used in deep learning to adjust the learning rate during training. The learning rate is a crucial hyperparameter that controls how quickly a model learns from the training data. A high learning rate can lead to fast convergence but may also cause the model to overshoot the optimal solution, while a low learning rate can lead to slow convergence. The ExponentialLR Scheduler helps to mitigate this issue by smoothly decaying the learning rate over time using an exponential decay schedule.

The exponential decay schedule is based on the formula $lr = lr_0 \times \gamma^{epoch}$ , where $lr_0$ is the initial learning rate, $\gamma$ is the decay rate, and $epoch$ is the current epoch. This schedule allows the learning rate to decrease rapidly at the beginning of training and then slow down as the model converges. The ExponentialLR Scheduler is commonly used in conjunction with the Stochastic Gradient Descent (SGD) optimizer, which is a widely used optimization algorithm in deep learning.

In the context of this problem, we need to understand how to create an ExponentialLR Scheduler and use it to simulate the decay of the learning rate over 10 epochs. We will also need to analyze the decay ratio and determine if the learning rate is decaying exponentially. This requires a basic understanding of PyTorch and its APIs for creating and using learning rate schedulers.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Create a PyTorch model and optimizer
Create an ExponentialLR Scheduler with the specified decay rate
Simulate the training process over 10 epochs, recording the learning rate at each epoch
Analyze the decay ratio and determine if the learning rate is decaying exponentially

This approach requires a basic understanding of PyTorch and its APIs for creating and using learning rate schedulers. We will also need to use Python to implement the simulation and analysis.

Step-by-Step Strategy

To solve this problem, we can follow these steps:

Import the necessary PyTorch modules and create a PyTorch model using nn.Linear(2, 1).
Create an SGD optimizer with the specified initial learning rate.
Create an ExponentialLR Scheduler with the specified decay rate.
Simulate the training process over 10 epochs, recording the learning rate at each epoch using a loop.
Calculate the decay ratio by dividing the last learning rate by the first learning rate.
Determine if the learning rate is decaying exponentially by comparing each learning rate to the previous one.

Common Pitfalls

Some common pitfalls to watch out for when implementing this solution include:

Forgetting to update the learning rate at each epoch
Using the wrong decay rate or initial learning rate
Failing to record the learning rate at each epoch
Incorrectly calculating the decay ratio or determining if the learning rate is decaying exponentially

Time & Space Complexity

The time complexity of this solution is O(n), where n is the number of epochs. This is because we need to simulate the training process over n epochs, and each epoch requires a constant amount of time to update the learning rate and record the result. The space complexity is O(n), as we need to store the learning rate at each epoch. However, since n is fixed at 10 in this problem, the time and space complexity are effectively constant.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.