Mini-Batch SGD
Implement one epoch of mini-batch Stochastic Gradient Descent for a simple linear model y=wâ‹…x+b.
Given training data, split it into mini-batches of size batch_size. For each batch, compute the mean gradient of MSE loss and update the parameters:
w=w−α⋅∂w∂L​,b=b−α⋅∂b∂L​
Process batches in order (first batch_size elements, then next batch_size, etc.). The last batch may be smaller.
Return the final (w, b) after one full epoch, rounded to 4 decimal places.
Example:
X = [1, 2, 3, 4] y = [2, 4, 6, 8] w = 0, b = 0 lr = 0.01, batch_size = 2
(0.5708, 0.1918)
- We split the training data into mini-batches of size
batch_size = 2, resulting in two batches:(X = [1, 2], y = [2, 4])and(X = [3, 4], y = [6, 8]). - For each batch, we compute the mean gradient of MSE loss. For the first batch, the predicted values are
$w \cdot X + b = 0 \cdot [1, 2] + 0 = [0, 0]$, and the gradients are$\frac{\partial L}{\partial w} = -2 \cdot ([2, 4] - [0, 0]) \cdot [1, 2] = -2 \cdot [2, 4] \cdot [1, 2] = -2 \cdot [2 + 8] = -20$and$\frac{\partial L}{\partial b} = -2 \cdot ([2, 4] - [0, 0]) = -2 \cdot [2 + 4] = -12$. We then update the parameters using these gradients and the learning rate$\alpha = 0.01$. - We repeat the process for the second batch, updating the parameters again.
- After processing both batches, we obtain the final
(w, b)values, which are then rounded to 4 decimal places, resulting in(0.5708, 0.1918).
Constraints:
- X: list of scalar inputs, y: list of targets
- w, b: initial scalar parameters
- learning_rate: float, batch_size: int >= 1
- Return (w, b) after one epoch, rounded to 4 decimal places
- MSE gradients: dw = -2/n * sum(x*(y-pred)), db = -2/n * sum(y-pred)
Background Knowledge
Stochastic Gradient Descent (SGD) is a fundamental algorithm in Machine Learning used for training linear and non-linear models. It's an optimization technique that minimizes the loss function by iteratively adjusting the model parameters in the direction of the negative gradient of the loss. In the context of linear regression, the model is defined as y=wâ‹…x+b, where w is the weight, b is the bias, x is the input, and y is the predicted output.
The Mean Squared Error (MSE) is a common loss function used for regression problems, which measures the average squared difference between predicted and actual values. The goal of SGD is to find the optimal values of w and b that minimize the MSE loss. The mini-batch variant of SGD splits the training data into smaller batches, computing the gradient of the loss for each batch, and updating the parameters accordingly. This approach helps to reduce the variance of the gradient estimates and improve the stability of the optimization process.
In the context of this problem, we need to implement one epoch of mini-batch SGD for a simple linear model. This involves splitting the training data into mini-batches, computing the mean gradient of the MSE loss for each batch, and updating the parameters w and b using the gradients and a learning rate α. The process is repeated for all batches, and the final values of w and b are returned after one full epoch.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.