SGD Parameter Update

easyFREEOptimization & TrainingDeep LearningDeep Learning 1

Problem Statement

Perform a single Stochastic Gradient Descent update step.

Background

In SGD, parameters are updated using gradients and a learning rate $\alpha$ (alpha):

$W = W - \alpha \cdot dW$

This simple rule moves parameters in the direction that reduces the loss.

Your Task

Write a function sgd_update(weights, gradients, learning_rate) that performs a single SGD update on a list of weights.

Output Format

Return a list of updated weights. Round each value to 4 decimal places.

Example

Input

weights=[1.0], gradients=[0.5], learning_rate=0.1

Output

[0.95]

Reasoning

Inputs: weights holds the current parameter values $W$ (here a single weight, $W = 1.0$ ); gradients holds $dW$ , the slope of the loss with respect to each weight (here $0.5$ ); learning_rate is the step size $\alpha = 0.1$ . The two lists line up position by position.
Step size: the gradient says the loss rises as $W$ grows, so the update moves $W$ the other way by $\alpha \cdot dW = 0.1 \times 0.5 = 0.05$ .
Update: $W_{\text{new}} = 1.0 - 0.05 = 0.95$ , rounded to 4 decimal places it stays $0.95$ , giving the output [0.95].
Output: a list the same length as the input, one updated value per weight, each depending only on its own gradient.
Notice that the sign of $dW$ decides the direction: a negative gradient makes the weight grow. With $W = 0$ , $dW = -1$ and $\alpha = 0.01$ the new weight is $0 - 0.01 \times (-1) = 0.01$ . A smaller $\alpha$ means a smaller, more cautious step.

Constraints

-1000 <= weights[i], gradients[i] <= 1000
0 < learning_rate <= 1
List length: 1 to 100 elements

Learn this in

Deep LearningCV Study Plan Linear RegressionML Study Plan Neural NetworksML Study Plan

More from Deep Learning 1

Early Stopping Checkeasy L2 Regularization Penaltyeasy Inverted Dropout Forwardmedium Binary Cross-Entropy Losseasy Momentum Update Stepmedium

Concept primer

1. Background Knowledge

Stochastic Gradient Descent (SGD) is a foundational optimization algorithm in machine learning for minimizing loss functions $L($ \mathbf{w} $)$ where $\mathbf{w}$ represents model parameters (weights). The core update rule is:

$\mathbf{w} \leftarrow \mathbf{w} - \alpha \nabla L(\mathbf{w})$

Here, $\alpha$ is the learning rate (step size), and $\nabla L($ \mathbf{w} $)$ is the gradient. In full-batch GD, the gradient uses all data; in SGD, it uses a single (or mini-batch) example, making it stochastic but computationally efficient for large datasets.

Key prerequisites:

Vectors/matrices: Weights $\mathbf{w} \in \mathbb{R}^n$ , gradients $\mathbf{dw} \in \mathbb{R}^n$
Element-wise operations: Update applies independently per parameter: $w_i \leftarrow w_i -$ \alpha $\cdot dw_i$
Numerical stability: Round to 4 decimals as specified to handle floating-point precision.

2. Algorithm Approach

The problem requires vanilla SGD (no momentum, adaptive rates like Adam, or variance reduction). Standard techniques:

Element-wise vector subtraction: Use NumPy broadcasting for efficiency.
Simultaneous update: Compute all new weights before overwriting (avoids using updated values prematurely).
Common libraries: numpy for vectorized operations (assumed available).

Pseudocode:

for each i in 0 to n-1:
 new_weights[i] = weights[i] - learning_rate * gradients[i]
return round(new_weights, 4)

3. Step-by-Step Strategy

Validate inputs: Ensure weights and gradients have identical length (1-100 elements per constraints).
Initialize output list: Create new list same size as input.
Compute updates: For each index $i$ , calculate $w_i' = w_i -$ \alpha $\cdot dw_i$ .
Apply rounding: Round each $w_i'$ to 4 decimal places using round(value, 4).
Return result: Output as Python list.

Sample Implementation:

def sgd_update(weights, gradients, learning_rate):
 if len(weights) != len(gradients):
 raise ValueError("Weights and gradients must have same length")
 updated = []
 for w, dw in zip(weights, gradients):
 new_w = w - learning_rate * dw
 updated.append(round(new_w, 4))
 return updated

Vectorized (NumPy) version:

import numpy as np
def sgd_update(weights, gradients, learning_rate):
 w = np.array(weights)
 dw = np.array(gradients)
 return np.round(w - learning_rate * dw, 4).tolist()

Verification: [1.0] - 0.1 * [0.5] = [0.95] ✓

4. Common Pitfalls

In-place updates: weights[i] -= alpha * gradients[i] uses partially updated values, causing errors in multi-parameter cases.
No rounding: Output [0.949999999] instead of [0.95] fails tests.
Integer division: Use float inputs/operations (Python 3 handles this).
Empty lists: Constraints guarantee 1+ elements, but edge case $n=1$ needs testing.
Sign error: Remember subtract gradient (negative direction reduces loss).
List vs. NumPy: Return Python list, not array.

Pitfall	Wrong	Correct
In-place	weights -=...	New list
Rounding	0.95	round(0.95, 4)
Type	np.array return	.tolist()

5. Time & Space Complexity

Time: $O(n)$ where $n =$ list length (1-100). Single pass with constant-time operations per element.
Space: $O(n)$ for output list + $O(n)$ temporary storage. In-place impossible without errors.

LaTeX notation: $\mathcal{O}(n)$ time, $\mathcal{O}(n)$ space.

This covers vanilla SGD parameter update completely—simple but fundamental to all deep learning optimizers!

solution.py

Python 3.13

Test results