ExponentialLR Scheduler

easyFREELearning Rate SchedulersPyTorch Advanced

Problem Statement

Use ExponentialLR for smooth exponential decay of the learning rate.

Background

ExponentialLR multiplies the learning rate by gamma every epoch: lr[n] = initial_lr * gamma^n.

Your Task

The starter code creates a model and SGD optimizer. Create an ExponentialLR scheduler with gamma=0.9.

The epoch loop and exponential verification are pre-filled.

Output Format

Returns a dictionary with "lr_history" (10 values), "decay_ratio", and "is_exponential".

Example

Input

None

Output

{'lr_history': [1.0, 0.9, 0.81, 0.729, 0.6561, 0.5905, 0.5314, 0.4783, 0.4305, 0.3874], 'decay_ratio': 0.3874, 'is_exponential': True}

Reasoning

There is no argument: the function builds a tiny linear model whose optimizer starts at a learning rate of $\eta_0 = 1.0$ , and the schedule uses decay factor $\gamma = 0.9$ .
Each epoch first records the current rate, then advances the schedule by one epoch, so the rate recorded at epoch $n$ is $\eta_n = \eta_0 \gamma^n$ .
Working it through: $\eta_1 = 0.9$ , $\eta_2 = 0.81$ , $\eta_3 = 0.729$ , $\eta_4 = 0.6561$ , and $\eta_5 = 0.59049 \to 0.5905$ once rounded to 4 decimal places. Then $0.5314$ , $0.4783$ , $0.4305$ , and $\eta_9 = 0.387420\ldots \to 0.3874$ .
Ten values cover $n = 0$ to $9$ , so the last entry is $\gamma^9$ , not $\gamma^{10}$ . That is why decay_ratio $= \eta_9 / \eta_0 = 0.3874$ .
Every consecutive ratio $\eta_n / \eta_{n-1}$ is $0.9$ (within $0.01$ even after rounding), so is_exponential is True.
Notice the curve: the rate shrinks by a constant fraction each epoch, so the absolute drop gets smaller over time ( $0.1$ at first, about $0.043$ at the end).

Constraints

gamma=0.9
Initial lr=1.0
10 epochs

More from PyTorch Advanced

Implement SGD with Momentumhard Implement Custom SGD Optimizerhard Weight Decay for Specific Layersmedium Freeze Backbone, Train Headmedium Per-Layer Learning Rateseasy

Concept primer

Background Knowledge

The ExponentialLR Scheduler is a type of learning rate scheduler used in deep learning to adjust the learning rate during training. The learning rate is a crucial hyperparameter that controls how quickly a model learns from the training data. A high learning rate can lead to fast convergence but may also cause the model to overshoot the optimal solution, while a low learning rate can lead to slow convergence. The ExponentialLR Scheduler helps to mitigate this issue by smoothly decaying the learning rate over time using an exponential decay schedule.

The exponential decay schedule is based on the formula $lr = lr_0 \times \gamma^{epoch}$ , where $lr_0$ is the initial learning rate, $\gamma$ is the decay rate, and $epoch$ is the current epoch. This schedule allows the learning rate to decrease rapidly at the beginning of training and then slow down as the model converges. The ExponentialLR Scheduler is commonly used in conjunction with the Stochastic Gradient Descent (SGD) optimizer, which is a widely used optimization algorithm in deep learning.

In the context of this problem, we need to understand how to create an ExponentialLR Scheduler and use it to simulate the decay of the learning rate over 10 epochs. We will also need to analyze the decay ratio and determine if the learning rate is decaying exponentially. This requires a basic understanding of PyTorch and its APIs for creating and using learning rate schedulers.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Create a PyTorch model and optimizer
Create an ExponentialLR Scheduler with the specified decay rate
Simulate the training process over 10 epochs, recording the learning rate at each epoch
Analyze the decay ratio and determine if the learning rate is decaying exponentially

This approach requires a basic understanding of PyTorch and its APIs for creating and using learning rate schedulers. We will also need to use Python to implement the simulation and analysis.

Step-by-Step Strategy

To solve this problem, we can follow these steps:

Import the necessary PyTorch modules and create a PyTorch model using nn.Linear(2, 1).
Create an SGD optimizer with the specified initial learning rate.
Create an ExponentialLR Scheduler with the specified decay rate.
Simulate the training process over 10 epochs, recording the learning rate at each epoch using a loop.
Calculate the decay ratio by dividing the last learning rate by the first learning rate.
Determine if the learning rate is decaying exponentially by comparing each learning rate to the previous one.

Common Pitfalls

Some common pitfalls to watch out for when implementing this solution include:

Forgetting to update the learning rate at each epoch
Using the wrong decay rate or initial learning rate
Failing to record the learning rate at each epoch
Incorrectly calculating the decay ratio or determining if the learning rate is decaying exponentially

Time & Space Complexity

The time complexity of this solution is O(n), where n is the number of epochs. This is because we need to simulate the training process over n epochs, and each epoch requires a constant amount of time to update the learning rate and record the result. The space complexity is O(n), as we need to store the learning rate at each epoch. However, since n is fixed at 10 in this problem, the time and space complexity are effectively constant.

solution.py

Python 3.13

Test results