Typorithm · hard

Positional Encoding

positional-encodingtransformerspositionalropeorder

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Positional Encoding

Tokenization & Embeddings

Self-attention is order-agnostic, so position must be injected. Classic models add sinusoidal or learned positional encodings to the embedding, while modern LLMs use RoPE (rotary embeddings), which rotates queries and keys by position to encode relative position and extrapolate to longer sequences.

Self-attention computes how much each token attends to every other, but it does so over an unordered set — shuffling the tokens would not change the raw attention math. So the model would not know word order unless we tell it.

Classic transformers solve this by adding a positional encoding vector — either fixed sinusoids or a learned per-position vector — to each token embedding before the first layer.

Modern LLMs favor RoPE (rotary position embedding): instead of adding a vector, it rotates the query and key vectors by an angle that depends on position. Because the attention score then depends on the relative angle between two tokens, RoPE captures relative position and extrapolates more gracefully to longer sequences.

Attention is order-agnostic; position must be injected.

Classic: add sinusoidal or learned encodings.

RoPE rotates Q/K to encode relative position.

// attention has no inherent order
const pe = positionalEncoding(pos, dModel);
const q = applyRoPE(query, pos);
const k = applyRoPE(key, pos);
// RoPE rotates Q/K by position

Read the implementation

Connect each part of the code to the concept before you type.

1. Why position is needed

Self-attention treats tokens as a set, so order must be added explicitly.

// attention has no inherent order

2. Classic encoding

Classic models compute a per-position vector (sinusoidal or learned) to add to the embedding.

const pe = positionalEncoding(pos, dModel);

3. RoPE on query

RoPE rotates the query vector by an angle determined by its position.

const q = applyRoPE(query, pos);

4. RoPE on key

The key is rotated the same way, so the dot product depends on relative position.

const k = applyRoPE(key, pos);

5. What RoPE does

RoPE encodes position by rotating queries and keys rather than adding a vector.

// RoPE rotates Q/K by position

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.