Single Layer Backpropagation
Compute the gradient of weights for a single linear layer.
Given input x, weights W, bias b, and the gradient of the loss with respect to the output ∂o∂L (called grad_output):
For o=W⋅x+b: ∂W∂L=grad_output⊗x(outer product) ∂b∂L=grad_output ∂x∂L=WT⋅grad_output
Return a tuple (grad_W, grad_b, grad_x) where grad_W is a 2D list and grad_b, grad_x are 1D lists. All rounded to 4 decimal places.
Example:
x = [1, 2] W = [[0.5, 0.3], [-0.2, 0.8]] grad_output = [1.0, 0.5]
([[1.0, 2.0], [0.5, 1.0]], [1.0, 0.5], [0.4, 0.7])
- We calculate ∂W∂L by taking the outer product of
grad_outputandx: ∂W∂L=[1.00.5]⊗[12]=[1.0∗10.5∗11.0∗20.5∗2]=[1.00.52.01.0] - Then, we calculate ∂b∂L which is simply
grad_output: ∂b∂L=[1.00.5] - Next, we calculate ∂x∂L by taking the dot product of WT and
grad_output: ∂x∂L=[0.50.3−0.20.8]⋅[1.00.5]=[0.5∗1.0+−0.2∗0.50.3∗1.0+0.8∗0.5]=[0.40.7] - The final output is (∂W∂L,∂b∂L,∂x∂L)=([1.00.52.01.0],[1.00.5],[0.40.7])
Constraints:
- x: 1D list (d_in), W: 2D list (d_out x d_in)
- grad_output: 1D list (d_out) - gradient from upstream
- Return (grad_W, grad_b, grad_x)
- grad_W: 2D list (d_out x d_in), grad_b: 1D (d_out), grad_x: 1D (d_in)
- Round to 4 decimal places
Background Knowledge
The problem revolves around backpropagation, a fundamental concept in neural networks. Backpropagation is an algorithm used to train neural networks by minimizing the loss function. It does this by computing the gradient of the loss with respect to the model's parameters, allowing the model to adjust its weights and biases to reduce the loss. In the context of this problem, we're focusing on a single linear layer, which is the simplest form of a neural network layer.
The key concepts here include the outer product (denoted by ⊗), which is used to compute the gradient of the weights, and matrix multiplication, which is essential for computing the output of the linear layer and the gradient of the input. Understanding how these operations work and how they're applied in the context of backpropagation is crucial for solving this problem. Additionally, familiarity with linear algebra concepts such as matrix transpose (denoted by WT) is necessary.
The problem also involves understanding the chain rule, a fundamental principle in calculus that allows us to compute the derivative of a composite function. In the context of backpropagation, the chain rule is used to propagate the gradient of the loss backwards through the network, adjusting the model's parameters at each step. This process is what enables neural networks to learn from their inputs and improve their performance over time.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.