v9.1.1
Menu
Sign In

Single Layer Backpropagation

mediumPRONeural NetworksMachine Learning 2

Compute the gradient of weights for a single linear layer.

Given input xx, weights WW, bias bb, and the gradient of the loss with respect to the output ∂L∂o\frac{\partial L}{\partial o} (called grad_output):

For o=W⋅x+bo = W \cdot x + b: ∂L∂W=grad_output⊗x(outer product)\frac{\partial L}{\partial W} = \text{grad\_output} \otimes x \quad (\text{outer product}) ∂L∂b=grad_output\frac{\partial L}{\partial b} = \text{grad\_output} ∂L∂x=WT⋅grad_output\frac{\partial L}{\partial x} = W^T \cdot \text{grad\_output}

Return a tuple (grad_W, grad_b, grad_x) where grad_W is a 2D list and grad_b, grad_x are 1D lists. All rounded to 4 decimal places.

Example

Input
x = [1, 2]
W = [[0.5, 0.3], [-0.2, 0.8]]
grad_output = [1.0, 0.5]
Output
([[1.0, 2.0], [0.5, 1.0]], [1.0, 0.5], [0.4, 0.7])
Reasoning
  • We calculate ∂L∂W\frac{\partial L}{\partial W} by taking the outer product of grad_output and x: ∂L∂W=[1.00.5]⊗[12]=[1.0∗11.0∗20.5∗10.5∗2]=[1.02.00.51.0]\frac{\partial L}{\partial W} = \begin{bmatrix} 1.0 \\ 0.5 \end{bmatrix} \otimes \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 1.0*1 & 1.0*2 \\ 0.5*1 & 0.5*2 \end{bmatrix} = \begin{bmatrix} 1.0 & 2.0 \\ 0.5 & 1.0 \end{bmatrix}
  • Then, we calculate ∂L∂b\frac{\partial L}{\partial b} which is simply grad_output: ∂L∂b=[1.00.5]\frac{\partial L}{\partial b} = \begin{bmatrix} 1.0 \\ 0.5 \end{bmatrix}
  • Next, we calculate ∂L∂x\frac{\partial L}{\partial x} by taking the dot product of WTW^T and grad_output: ∂L∂x=[0.5−0.20.30.8]⋅[1.00.5]=[0.5∗1.0+−0.2∗0.50.3∗1.0+0.8∗0.5]=[0.40.7]\frac{\partial L}{\partial x} = \begin{bmatrix} 0.5 & -0.2 \\ 0.3 & 0.8 \end{bmatrix} \cdot \begin{bmatrix} 1.0 \\ 0.5 \end{bmatrix} = \begin{bmatrix} 0.5*1.0 + -0.2*0.5 \\ 0.3*1.0 + 0.8*0.5 \end{bmatrix} = \begin{bmatrix} 0.4 \\ 0.7 \end{bmatrix}
  • The final output is (∂L∂W,∂L∂b,∂L∂x)=([1.02.00.51.0],[1.00.5],[0.40.7])(\frac{\partial L}{\partial W}, \frac{\partial L}{\partial b}, \frac{\partial L}{\partial x}) = (\begin{bmatrix} 1.0 & 2.0 \\ 0.5 & 1.0 \end{bmatrix}, \begin{bmatrix} 1.0 \\ 0.5 \end{bmatrix}, \begin{bmatrix} 0.4 \\ 0.7 \end{bmatrix})

Constraints

  • x: 1D list (d_in), W: 2D list (d_out x d_in)
  • grad_output: 1D list (d_out) - gradient from upstream
  • Return (grad_W, grad_b, grad_x)
  • grad_W: 2D list (d_out x d_in), grad_b: 1D (d_out), grad_x: 1D (d_in)
  • Round to 4 decimal places

Learn this in

More from Machine Learning 2

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Test results

Sign in to run your code against the test cases.