Feed-Forward Network

easyFREETransformer ComponentsLLM 1: Foundations

Implement a transformer feed-forward network (FFN).

The FFN applies two linear transformations with a ReLU activation in between: $\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$

For simplicity, use no biases (b1=0, b2=0). Use numpy seed 42 to generate W1 and W2.

Input: ffn(d, d_ff, x)

d: model dimension
d_ff: FFN hidden dimension
x: NumPy input vector of shape (d,)

Output: Return the output vector of shape (d,); it is printed rounded to 4 decimal places.

Generate weights: np.random.seed(42), W1 = np.random.randn(d, d_ff) * 0.1, W2 = np.random.randn(d_ff, d) * 0.1

Example

Input

2 4
1.0 -1.0

Output

-0.0169 -0.0007

Reasoning

Inputs: $d = 2$ , $d_{ff} = 4$ and $x = [1, -1]$ . So $W_1$ is $2 \times 4$ (expands the vector to 4 hidden units) and $W_2$ is $4 \times 2$ (projects back to 2), both drawn from the seeded standard normal and scaled by $0.1$ .
First layer: with $x = [1, -1]$ , hidden unit $k$ is (row 1 of $W_1$ ) minus (row 2), e.g. $h_1 = 0.0497 - (-0.0234) = 0.0731$ . All four: $[0.0731,\ 0.0096,\ -0.0932,\ 0.0756]$ .
ReLU replaces the negative unit with $0$ : $[0.0731,\ 0.0096,\ 0,\ 0.0756]$ .
Second layer: output $j$ is $\sum_k h_k (W_2)_{kj}$ . For $j = 1$ : $-0.00343 - 0.00044 + 0 - 0.01303 = -0.01691$ ; for $j = 2$ : $0.00397 - 0.00045 + 0 - 0.00425 = -0.00073$ .
Output: -0.0169 -0.0007, a vector of length $d = 2$ again, rounded to 4 decimal places.
Notice the third hidden unit contributes nothing, and an all-zero input gives an all-zero output since there are no biases.

Constraints

1 <= d <= 10, 1 <= d_ff <= 20
Use np.random.seed(42) for weight initialization
Scale weights by 0.1
ReLU: max(0, x)
Round to 4 decimal places

Learn this in

Transformer ArchitectureLLM Study Plan Attention MechanismsNLP Study Plan

More from LLM 1: Foundations

Transformer Block Forward Passhard Layer Normalizationeasy KV-Cache Attentionhard Multi-Head Attentionmedium Causal Attention Maskmedium

Concept primer

Background Knowledge

The Transformer architecture, introduced in the paper "Attention is All You Need" by Vaswani et al., revolutionized the field of Natural Language Processing (NLP). One of the key components of the Transformer is the Feed-Forward Network (FFN), which is used in the encoder and decoder layers. The FFN is a simple yet powerful module that applies two linear transformations with a ReLU activation function in between. This allows the model to learn complex representations of the input data.

The FFN is defined as $\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$ , where $x$ is the input vector, $W_1$ and $W_2$ are learnable weights, and $b_1$ and $b_2$ are biases. In this problem, we are asked to implement a simplified version of the FFN, where the biases are set to zero. The weights $W_1$ and $W_2$ are generated using np.random.seed(42) to ensure reproducibility.

The ReLU activation function, also known as the rectified linear unit, is a widely used activation function in deep neural networks. It is defined as $\text{ReLU}(x) = \max(0, x)$ , which means that all negative values are set to zero, and all positive values are left unchanged. The ReLU activation function is used to introduce non-linearity into the model, allowing it to learn more complex representations of the input data.

Algorithm/Approach

The general approach to solving this problem is to follow the definition of the FFN and apply the two linear transformations with a ReLU activation function in between. The key steps involve:

Generating the weights $W_1$ and $W_2$ using np.random.seed(42)
Applying the first linear transformation to the input vector $x$
Applying the ReLU activation function to the result
Applying the second linear transformation to the result

Step-by-Step Strategy

To implement the solution, follow these steps:

Read the input values d and d_ff from the first line.
Read the input vector x from the second line.
Generate the weights W1 and W2 using np.random.seed(42).
Apply the first linear transformation to the input vector x using W1.
Apply the ReLU activation function to the result.
Apply the second linear transformation to the result using W2.
Round the output vector to 4 decimal places.

Common Pitfalls

Some common pitfalls to watch out for when implementing the solution include:

Forgetting to set the biases to zero
Using the wrong seed value for generating the weights
Applying the ReLU activation function incorrectly
Not rounding the output vector to 4 decimal places

Time & Space Complexity

The time complexity of the solution is O(d * d_ff), where d is the model dimension and d_ff is the FFN hidden dimension. The space complexity is also O(d * d_ff), as we need to store the weights W1 and W2. The output vector has a size of d, so the space complexity is dominated by the weights.

Here is a simple code snippet to generate the weights:

import numpy as np

# Set the seed value
np.random.seed(42)

# Define the model dimension and FFN hidden dimension
d = 10 # example value
d_ff = 20 # example value

# Generate the weights W1 and W2
W1 = np.random.randn(d, d_ff) * 0.1
W2 = np.random.randn(d_ff, d) * 0.1

solution.py

Python 3.13

Test results