MLP Forward Pass

mediumPRONeural NetworksMachine Learning 2

Implement the forward pass of a two-layer Multi-Layer Perceptron (MLP).

Given input $x$ , two weight matrices $W_1$ , $W_2$ and two bias vectors $b_1$ , $b_2$ :

Hidden layer: $h = \text{ReLU}(W_1 \cdot x + b_1)$
Output layer: $o = W_2 \cdot h + b_2$

Where $\text{ReLU}(z) = \max(0, z)$ applied element-wise.

Return the output vector, rounded to 4 decimal places.

Example

Input

x = [1, 2]
W1 = [[0.5, 0.3], [-0.2, 0.8]]
b1 = [0.1, -0.1]
W2 = [[0.4, -0.5]]
b2 = [0.2]

Output

[0.03]

Reasoning

Inputs: $x = [1, 2]$ is a vector with $d_{in} = 2$ features. $W_1$ is $2 \times 2$ , so the hidden layer has $d_{hidden} = 2$ units; $W_2$ is $1 \times 2$ , so there is a single output ( $d_{out} = 1$ ).
Hidden pre-activations: each hidden unit takes the dot product of its row of $W_1$ with $x$ , plus its bias. Unit 1: $0.5 \cdot 1 + 0.3 \cdot 2 + 0.1 = 1.2$ . Unit 2: $-0.2 \cdot 1 + 0.8 \cdot 2 - 0.1 = 1.3$ .
ReLU: both values are positive, so $h = [1.2, 1.3]$ passes through unchanged.
Output layer: $o = 0.4 \cdot 1.2 + (-0.5) \cdot 1.3 + 0.2 = 0.48 - 0.65 + 0.2 = 0.03$ , with no activation applied.
Result: one output unit gives a length-1 list, [0.03] after rounding to 4 decimal places.
Notice that ReLU only matters when a pre-activation is negative: with $x = [2]$ and $W_1 = [[-1], [1]]$ , the first hidden unit is $-2$ and gets zeroed, so only the second unit (value $2$ ) reaches the output.

Constraints

x: 1D list (input vector, d_in features)
W1: 2D list (d_hidden x d_in), b1: 1D list (d_hidden)
W2: 2D list (d_out x d_hidden), b2: 1D list (d_out)
Return 1D list of output values rounded to 4 decimal places

Learn this in

Transformer ArchitectureLLM Study Plan Neural NetworksML Study Plan Vision-Language AlignmentVLM Study Plan

More from Machine Learning 2

Kernel PCAhard PCA via Covariance Matrixmedium Variance Explained Ratioeasy Feature Scalingeasy GAN Discriminator Losshard

Concept primerPREVIEW

Background Knowledge

The Multi-Layer Perceptron (MLP) is a type of neural network that consists of multiple layers of interconnected nodes or "neurons." Each layer receives input from the previous layer, performs a computation on that input, and then sends the output to the next layer. The forward pass refers to the process of passing input through the network, layer by layer, to produce an output. In this problem, we're dealing with a two-layer MLP, which means we have an input layer, a hidden layer, and an output layer.

The hidden layer uses the ReLU (Rectified Linear Unit) activation function, which applies the $\max(0, z)$ function element-wise to the input. This means that any negative values in the input will be set to 0, while positive values will remain unchanged. The ReLU function is commonly used in neural networks because it's simple to compute and helps to introduce non-linearity into the model. The output layer uses a linear activation function, which means that the output is simply a weighted sum of the inputs, plus a bias term.

In order to implement the forward pass of an MLP, we need to have a good understanding of linear algebra and matrix operations. Specifically, we need to know how to perform matrix multiplication and addition, as well as how to apply element-wise functions like ReLU. We also need to understand how to represent the weights and biases of the network as matrices and vectors, and how to use these representations to compute the output of each layer.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Compute the output of the hidden layer by multiplying the input by the weight matrix, adding the bias vector, and applying the ReLU activation function.
Compute the output of the output layer by multiplying the hidden layer output by the weight matrix, adding the bias vector, and applying the linear activation function.
Return the output of the output layer, rounded to 4 decimal places.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results