📘

Feed-Forward Network

Implement a transformer feed-forward network (FFN).

The FFN applies two linear transformations with a ReLU activation in between: $\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$

For simplicity, use no biases (b1=0, b2=0). Use numpy seed 42 to generate W1 and W2.

Input:

Output: Output vector of dimension d, rounded to 4 decimal places.

Generate weights: np.random.seed(42), W1 = np.random.randn(d, d_ff) * 0.1, W2 = np.random.randn(d_ff, d) * 0.1

Input:

2 4
1.0 -1.0

Output:

0.0107 -0.0510

Reasoning:

We generate the weights $W_1$ and $W_2$ using np.random.seed(42), with $W_1 = np.random.randn(2, 4) * 0.1$ and $W_2 = np.random.randn(4, 2) * 0.1$ .
The input vector $x = [1.0, -1.0]$ is then transformed by $W_1$ and passed through the ReLU activation function: $ReLU(xW_1) = ReLU([1.0, -1.0] \cdot W_1)$ .
The result is then transformed by $W_2$ : $ReLU(xW_1)W_2 = ReLU([1.0, -1.0] \cdot W_1) \cdot W_2$ .
The final output is the result of this transformation, rounded to 4 decimal places: $[0.0107, -0.0510]$ .

Editor

Python 3.13.1

0/0

Run code to see test results.

📘