📘

Rotary Position Embedding

Implement Rotary Position Embedding (RoPE).

RoPE rotates pairs of embedding dimensions by position-dependent angles. For a vector x of dimension D at position pos:

Split x into pairs: (x[0], x[1]), (x[2], x[3]), ...
For pair i, compute angle θ_i = pos / 10000^(2i/D)
Apply rotation: x'[2i] = x[2i] * cos(θ_i) - x[2i+1] * sin(θ_i) x'[2i+1] = x[2i] * sin(θ_i) + x[2i+1] * cos(θ_i)

Input:

Output: Rotated vector, values rounded to 4 decimal places.

Input:

1 4
1.0 0.0 1.0 0.0

Output:

0.5403 0.8415 0.9999 0.0100

Reasoning:

The input vector is split into pairs: $(1.0, 0.0)$ and $(1.0, 0.0)$ .
For each pair $i$ , the angle $\theta_i$ is computed: $\theta_0 = \frac{1}{10000^{2/4}} = \frac{1}{100}$ and $\theta_1 = \frac{1}{10000^{4/4}} = \frac{1}{10000}$ .
The rotation is applied to each pair:
- For the first pair: $x'[0] = 1.0 \cdot \cos(\frac{1}{100}) - 0.0 \cdot \sin(\frac{1}{100})$ and $x'[1] = 1.0 \cdot \sin(\frac{1}{100}) + 0.0 \cdot \cos(\frac{1}{100})$ , resulting in approximately $0.5403$ and $0.8415$ respectively,
- For the second pair: $x'[2] = 1.0 \cdot \cos(\frac{1}{10000}) - 0.0 \cdot \sin(\frac{1}{10000})$ and $x'[3] = 1.0 \cdot \sin(\frac{1}{10000}) + 0.0 \cdot \cos(\frac{1}{10000})$ , resulting in approximately $0.9999$ and $0.0100$ respectively.
The final output is the rotated vector: $0.5403, 0.8415, 0.9999, 0.0100$ .

Editor

Python 3.13.1

0/0

Run code to see test results.

📘