Implement the sinusoidal positional encoding from "Attention Is All You Need".

For a sequence of length L and model dimension D, compute the positional encoding matrix PE where:

Input:

Output: The L x D positional encoding matrix, values rounded to 4 decimal places, one row per position.

[ 0.0000  1.0000  0.0000  1.0000]
[ 0.8415  0.5403  0.0100  0.9999]

We calculate the positional encoding matrix PE for a sequence of length L = 2 and model dimension D = 4.
For each position pos in the sequence, we compute the values of PE(pos, 2i) and PE(pos, 2i+1) using the given formulas: $PE(pos, 2i) = \sin(pos / 10000^{2i/D})$ and $PE(pos, 2i+1) = \cos(pos / 10000^{2i/D})$ .
We evaluate these formulas for pos = 0 and pos = 1, and i = 0 and i = 1, to get the values for the first and second positions:
- For pos = 0, we get $PE(0, 0) = \sin(0 / 10000^{0/4}) = \sin(0) = 0$ , $PE(0, 1) = \cos(0 / 10000^{0/4}) = \cos(0) = 1$ , $PE(0, 2) = \sin(0 / 10000^{1/4}) = \sin(0) = 0$ , $PE(0, 3) = \cos(0 / 10000^{1/4}) = \cos(0) = 1$ .
- For pos = 1, we get $PE(1, 0) = \sin(1 / 10000^{0/4}) = \sin(1) \approx 0.8415$ , $PE(1, 1) = \cos(1 / 10000^{0/4}) = \cos(1) \approx 0.5403$ , $PE(1, 2) = \sin(1 / 10000^{1/4}) \approx \sin(0.0100) \approx 0.0100$ , $PE(1, 3) = \cos(1 / 10000^{1/4}) \approx \cos(0.0100) \approx 0.9999$ .
The final output is the matrix with these computed values, rounded to 4 decimal places: [ 0.0000 1.0000

Editor

Python 3.13.1

0/0

Run code to see test results.

📘

Sinusoidal Positional Encoding