📘

Feed-Forward Network

EasyTransformer Components 📚 LLM 1: Foundations

Implement a transformer feed-forward network (FFN).

The FFN applies two linear transformations with a ReLU activation in between: $\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$

For simplicity, use no biases (b1=0, b2=0). Use numpy seed 42 to generate W1 and W2.

Input:

Line 1: d d_ff (model dimension, FFN hidden dimension)
Line 2: space-separated floats (input vector of dimension d)

Output: Output vector of dimension d, rounded to 4 decimal places.

Generate weights: np.random.seed(42), W1 = np.random.randn(d, d_ff) * 0.1, W2 = np.random.randn(d_ff, d) * 0.1

Example:

Input:

2 4
1.0 -1.0

Output:

0.0107 -0.0510

Reasoning:

We generate the weights $W_1$ and $W_2$ using np.random.seed(42), with $W_1 = np.random.randn(2, 4) * 0.1$ and $W_2 = np.random.randn(4, 2) * 0.1$ .
The input vector $x = [1.0, -1.0]$ is then transformed by $W_1$ and passed through the ReLU activation function: $ReLU(xW_1) = ReLU([1.0, -1.0] \cdot W_1)$ .
The result is then transformed by $W_2$ : $ReLU(xW_1)W_2 = ReLU([1.0, -1.0] \cdot W_1) \cdot W_2$ .
The final output is the result of this transformation, rounded to 4 decimal places: $[0.0107, -0.0510]$ .

Constraints:

1 <= d <= 10, 1 <= d_ff <= 20
Use np.random.seed(42) for weight initialization
Scale weights by 0.1
ReLU: max(0, x)
Round to 4 decimal places

More from LLM 1: Foundations

Transformer Block Forward PassHard Layer NormalizationEasy KV-Cache AttentionHard Multi-Head AttentionMedium Causal Attention MaskMedium

Background Knowledge

The Transformer architecture, introduced in the paper "Attention is All You Need" by Vaswani et al., revolutionized the field of Natural Language Processing (NLP). One of the key components of the Transformer is the Feed-Forward Network (FFN), which is used in the encoder and decoder layers. The FFN is a simple yet powerful module that applies two linear transformations with a ReLU activation function in between. This allows the model to learn complex representations of the input data.

The FFN is defined as $\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$ , where $x$ is the input vector, $W_1$ and $W_2$ are learnable weights, and $b_1$ and $b_2$ are biases. In this problem, we are asked to implement a simplified version of the FFN, where the biases are set to zero. The weights $W_1$ and $W_2$ are generated using np.random.seed(42) to ensure reproducibility.

The ReLU activation function, also known as the rectified linear unit, is a widely used activation function in deep neural networks. It is defined as $\text{ReLU}(x) = \max(0, x)$ , which means that all negative values are set to zero, and all positive values are left unchanged. The ReLU activation function is used to introduce non-linearity into the model, allowing it to learn more complex representations of the input data.

Algorithm/Approach

The general approach to solving this problem is to follow the definition of the FFN and apply the two linear transformations with a ReLU activation function in between. The key steps involve:

Generating the weights $W_1$ and $W_2$ using np.random.seed(42)
Applying the first linear transformation to the input vector $x$
Applying the ReLU activation function to the result
Applying the second linear transformation to the result

Step-by-Step Strategy

To implement the solution, follow these steps:

Read the input values d and d_ff from the first line.
Read the input vector x from the second line.
Generate the weights W1 and W2 using np.random.seed(42).
Apply the first linear transformation to the input vector x using W1.
Apply the ReLU activation function to the result.
Apply the second linear transformation to the result using W2.
Round the output vector to 4 decimal places.

Common Pitfalls

Some common pitfalls to watch out for when implementing the solution include:

Forgetting to set the biases to zero
Using the wrong seed value for generating the weights
Applying the ReLU activation function incorrectly
Not rounding the output vector to 4 decimal places

Time & Space Complexity

The time complexity of the solution is O(d * d_ff), where d is the model dimension and d_ff is the FFN hidden dimension. The space complexity is also O(d * d_ff), as we need to store the weights W1 and W2. The output vector has a size of d, so the space complexity is dominated by the weights.

Here is a simple code snippet to generate the weights:

import numpy as np

# Set the seed value
np.random.seed(42)

# Define the model dimension and FFN hidden dimension
d = 10 # example value
d_ff = 20 # example value

# Generate the weights W1 and W2
W1 = np.random.randn(d, d_ff) * 0.1
W2 = np.random.randn(d_ff, d) * 0.1

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.