Sinusoidal Positional Encoding

mediumFREEEmbeddings & Positional EncodingLLM 1: Foundations

Implement the sinusoidal positional encoding from "Attention Is All You Need".

For a sequence of length L and model dimension D, compute the positional encoding matrix PE where:

PE(pos, 2i) = sin(pos / 10000^(2i/D))
PE(pos, 2i+1) = cos(pos / 10000^(2i/D))

Input:

L: sequence length (number of positions)
D: model dimension (even)

Output: Implement sinusoidal_pe(L, D) returning the L x D positional encoding matrix as a NumPy array, one row per position. The tests print each value to 4 decimal places.

Example

Input

2 4

Output

[0.0000 1.0000 0.0000 1.0000]
[0.8415 0.5403 0.0100 1.0000]

Reasoning

Inputs: L $= 2$ positions (rows 0 and 1) and D $= 4$ dimensions, so the result is a $2 \times 4$ matrix. The 4 columns form 2 sin/cos pairs, one pair per frequency.
Denominators: pair $i = 0$ (columns 0–1) uses $10000^{0/4} = 1$ ; pair $i = 1$ (columns 2–3) uses $10000^{2/4} = 100$ .
pos = 0: every angle is $0$ , so each pair gives $\sin 0 = 0$ , $\cos 0 = 1$ → [0.0000 1.0000 0.0000 1.0000]. Position 0 always looks like this, whatever $D$ is.
pos = 1, pair 0: angle $1/1 = 1$ radian → $\sin 1 \approx 0.8415$ , $\cos 1 \approx 0.5403$ .
pos = 1, pair 1: angle $1/100 = 0.01$ → $\sin 0.01 \approx 0.0099998 \to 0.0100$ and $\cos 0.01 \approx 0.99995 \to 1.0000$ after rounding to 4 decimal places.
Notice that the first pair swings a full radian per step while the second barely moves: higher pairs rotate more slowly, which is how the encoding distinguishes positions far apart.

Constraints

1 <= L <= 20, 2 <= D <= 16 (D is even)
Use base 10000 for the frequency
Round to 4 decimal places

Learn this in

Tokenization & EmbeddingsLLM Study Plan Attention MechanismsNLP Study Plan

More from LLM 1: Foundations

Transformer Block Forward Passhard Feed-Forward Networkeasy Layer Normalizationeasy KV-Cache Attentionhard Multi-Head Attentionmedium

Concept primer

Background Knowledge

The concept of positional encoding is crucial in sequence-to-sequence models, particularly in the Transformer architecture, as it allows the model to understand the position of each element in the input sequence. This is necessary because the self-attention mechanism, a key component of the Transformer, is permutation-invariant, meaning it does not inherently capture the order of the input elements. The sinusoidal positional encoding is one method to incorporate positional information into the input embeddings.

The idea behind sinusoidal positional encoding is to generate a set of vectors that can be added to the input embeddings to convey positional information. These vectors are designed such that they can be learned by the model and are fixed during training, meaning they do not have any learnable parameters. The formula provided in the problem description, $PE(pos, 2i) = \sin(pos / 10000^{2i/D})$ and $PE(pos, 2i+1) = \cos(pos / 10000^{2i/D})$ , is used to compute the positional encoding matrix $PE$ , where $pos$ is the position in the sequence, $i$ is the dimension, $D$ is the model dimension, and $L$ is the sequence length.

Understanding the mathematical properties of the sine and cosine functions is essential for grasping how this encoding works. The use of $10000^{2i/D}$ as the base for the exponential decay allows the model to capture a wide range of positional relationships at different scales. This encoding scheme enables the model to effectively capture long-range dependencies in the input sequence, which is a critical aspect of many natural language processing tasks.

Algorithm/Approach

The general approach to solving this problem involves understanding the mathematical formula provided and implementing it in a programming language. The key steps involve iterating over each position in the sequence and each dimension in the model, applying the given formulas to compute the positional encoding values. This requires a basic understanding of nested loops, array or matrix operations, and trigonometric functions.

Step-by-Step Strategy

To implement the solution:

Read the input values for sequence length $L$ and model dimension $D$ .
Initialize an $L \times D$ matrix to store the positional encoding values.
Iterate over each position $pos$ from 0 to $L-1$ .
For each position, iterate over each dimension $i$ from 0 to $D-1$ .
Apply the formulas $PE(pos, 2i) = \sin(pos / 10000^{2i/D})$ and $PE(pos, 2i+1) = \cos(pos / 10000^{2i/D})$ to compute the positional encoding values.
Store the computed values in the corresponding positions in the matrix.
Round the values in the matrix to 4 decimal places.
Output the matrix, with one row per position.

Common Pitfalls

Incorrect implementation of the formulas, especially regarding the indexing and the application of the exponential decay.
Failure to round the values to the specified precision.
Inadequate handling of the input values, such as not checking for valid sequence lengths and model dimensions.

Time & Space Complexity

Time Complexity: The time complexity of this algorithm is $O(L \times D)$ , where $L$ is the sequence length and $D$ is the model dimension. This is because for each position in the sequence, we are iterating over each dimension.
Space Complexity: The space complexity is also $O(L \times D)$ , as we need to store the positional encoding matrix of size $L \times D$ .

solution.py

Python 3.13

Test results