Simulate Sinusoidal Positional Encoding

mediumFREETransformersMath, ArrayAdvanced Architectures

Positional Encoding injects positional information into Transformer input sequences, as self-attention is inherently permutation invariant.

The sinusoidal positional encoding for token at position $i$ with embedding dimension $D$ is calculated as:

$p_{i,k} = \begin{cases} \sin(i / 10000^{k/D}) & \text{if } k \text{ is even} \\ \cos(i / 10000^{(k-1)/D}) & \text{if } k \text{ is odd} \end{cases}$

Task: Implement a function that generates the positional encoding vector $p$ of length $D$ for a token at sequence index $i$ .

Example

Input

i=0, D=4

Output

[0.0000, 1.0000, 0.0000, 1.0000]

Reasoning

Inputs: i is the token's position in the sequence (here $i = 0$ , the first token) and D is the embedding size (here $D = 4$ ), so the output is a vector of 4 numbers indexed $k = 0, 1, 2, 3$ .
Pairing: dimensions come in sine/cosine pairs that share one frequency. For $D = 4$ : pair $(k=0, k=1)$ divides by $10000^{0/4} = 1$ , and pair $(k=2, k=3)$ divides by $10000^{2/4} = 100$ .
Angles: every angle is $i$ divided by that scale, so at $i = 0$ all angles are $0/1 = 0$ and $0/100 = 0$ .
Values: even slots take the sine of 0, which is $0$ , and odd slots take the cosine of 0, which is $1$ , producing [0.0000, 1.0000, 0.0000, 1.0000].
Notice: position 0 hides the frequencies entirely. At $i = 1$ the first pair becomes sine and cosine of $1$ radian ( $0.8415$ , $0.5403$ ), while the slow pair barely moves: sine and cosine of $0.01$ give $0.01$ and $1.0$ . Later dimensions rotate more slowly, which is what lets distant positions stay distinguishable.

Constraints

$0 \leq i < 1000$
$D$ is an even integer: $2 \leq D \leq 512$

Learn this in

NLP FundamentalsML Study Plan

More from Advanced Architectures

Complete Mixer Block with Residualshard Channel Mixing with GELUmedium Token Mixing Layermedium Bidirectional Scan for Visionmedium Discretize Continuous State Space Modelhard

Concept primer

1. Background Knowledge

Positional Encoding addresses Transformer's permutation invariance: self-attention treats all tokens equally regardless of order, lacking inherent sequential awareness. The original Transformer uses sinusoidal positional encoding to inject position $i$ information into $D$ -dimensional embeddings via periodic sine/cosine functions with increasing frequencies $10000^{k/D}$ , creating unique, wavelength-diverse encodings. This deterministic approach ensures fixed-length extrapolation and relative position learning via phase differences.

Prerequisites: Basic trigonometry ( $\sin$ , $\cos$ ), array indexing (0-based), floating-point math. $D$ even ensures paired sin/cos dimensions.

2. Algorithm Approach

Direct formula implementation: for each dimension $k \in [0, D-1]$ ,

p_{i,k} = \begin{cases} \sin\left(i / 10000^{2k/D}\right) & k \text{ even} \\ \cos\left(i / 10000^{2(k-1)/D}\right) & k \text{ odd} \end{cases}

Key insight: Even indices use $\sin$ with frequency scaling $2k/D$ ; odd use $\cos$ shifted by one pair $(k-1)$ . Precompute powers via log(10000) * k/D for numerical stability.

Common technique: Vectorized NumPy/PyTorch loops or broadcasting over $k$ .

3. Step-by-Step Strategy

Initialize zero vector $p$ of shape $(D,)$ .
Loop over dimensions $k = 0$ to $D-1$ :

Compute angular frequency: $\omega_k = 10000^{-k/D}$ (use exp(-log(10000) * k/D)).
If $k \% 2 == 0$ : $p_k = \sin(i \cdot$ \omega $_k)$ .
Else: $p_k = \cos(i \cdot$ \omega $_{k-1})$ where $\omega_{k-1} = 10000^{-(k-1)/D}$ .

Return $p$ .

Vectorized Python example:

import numpy as np

def positional_encoding(i: int, D: int) -> np.ndarray:
 p = np.zeros(D)
 div_term = np.exp(-np.log(10000.0) * np.arange(0, D, 2) / D) # Even freqs
 p[0::2] = np.sin(i * div_term) # Even: sin(i / 10000^{k/D})
 p[1::2] = np.cos(i * np.roll(div_term, -1)) # Odd: cos(i / 10000^{(k-1)/D})
 return p[:D] # Truncate if needed

Note: np.roll(div_term, -1) shifts for odd indices; slice [:D] handles even $D$ .

4. Common Pitfalls

Off-by-one indexing: Odd $k$ uses $(k-1)/D$ , not $k/D$ —easy mix-up.
Numerical instability: Direct 10000 ** (k/D) overflows; use exp(log(10000) * k/D).
Even $D$ assumption: Code fails if $D$ odd (last dim unpaired); constraints guarantee even.
Integer division: Use float exponents (k/D, not k//D).
Range errors: $i < 1000$ , $D \leq 512$ safe, but verify no overflow in sin/cos args.
1-based vs 0-based: Formula uses 0-based $i \geq 0$ .

5. Time & Space Complexity

Time: $O(D)$ per call—single pass over dimensions with constant-time trig ops.
Space: $O(D)$ for output vector + $O(D/2)$ temp array; optimal as output required.
Batch optimization: For sequence length $N$ , precompute matrix $O(ND)$ ; here single $i$ .

This yields exact original Transformer encoding, enabling position-aware attention. Test: positional_encoding(0, 4) → $[0, 1, 0, 1]$ .

solution.py

Python 3.13

Test results