Why it matters
Attention compares content, not order: the same tokens in a different sequence produce the same pairwise scores. Language, code and retrieval all depend on order, so the model needs a signal that distinguishes position one from position ten. The design question is whether to encode an absolute index, a relative distance, or a transformation of the vectors themselves, because that choice decides how the model behaves beyond its trained context length.
Mental model
Positional encoding is seat numbering in a theatre: the seats are identical, so without numbers nobody knows who is in front. The metaphor breaks because the numbers are not labels the model reads — they are added to or rotated into the content vectors, so position and meaning are entangled in the same space.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.
- d_model
- 12
- dimensions shown
- d₀ – d₄
- parameters
- 0 · fixed formula
Step 1 of 4
Position 0
Every sine is 0.000 and every cosine is 1.000 at position zero. The row is determined by the formula, not fitted, so the first position looks the same in every model that uses this scheme.
Next: Position 1
Figure. The first five dimensions of a sinusoidal encoding for positions 0 to 7 at d_model = 12. Each row is one position's vector and each column is a sine or cosine of that position at a fixed frequency, so the table is a formula rather than a learned table and every entry can be recomputed. Shade is one signed scale from −1.000 to +1.000; the printed value carries the sign.
What to notice. Column d₀ is sin(pos) and completes a full cycle in these eight rows; d₂ divides by 10000^(2/12) ≈ 4.64 and turns more slowly; d₄ is the slowest shown and moves only from 0.000 to 0.319. The formula is defined at any index, which is what 'extrapolates in principle' means, but a model has only learned to use the range its training covered. The figure shows five of twelve dimensions and no learned or rotary scheme, so it cannot rank them.
Text version
An eight by five matrix. Rows are positions 0 to 7 and columns dimensions d₀ to d₄ of a 12-dimensional sinusoidal encoding. Row 0 prints 0.000, 1.000, 0.000, 1.000 and 0.000; row 1 prints 0.841, 0.540, 0.214, 0.977 and 0.046; row 2 prints 0.909, −0.416, 0.418, 0.909 and 0.093; row 3 prints 0.141, −0.990, 0.602, 0.798 and 0.139; row 4 prints −0.757, −0.654, 0.759, 0.651 and 0.185; row 5 prints −0.959, 0.284, 0.881, 0.474 and 0.230; row 6 prints −0.279, 0.960, 0.962, 0.275 and 0.275; row 7 prints 0.657, 0.754, 0.998, 0.063 and 0.319. Shade runs from teal at −1.000 through zero to accent at +1.000. Steps highlight position 0, position 1, the d₀ column across all eight positions, and the fast d₃ column against the slow d₄ column. Hovering a cell reports the position, the dimension and the printed value.
- Decide whether the model needs absolute position, relative distance, or both; the choice constrains the attention implementation.
- For fixed sinusoids, compute sine and cosine at geometrically spaced frequencies and add them to the input embeddings.
- For learned absolute encodings, allocate an embedding row per index and accept that indices beyond the table have no representation.
- For relative schemes, add a learned bias to attention scores based on the distance between query and key positions.
- For RoPE, rotate query and key vectors by an angle proportional to position, so their dot product depends on the offset.
- Validate length generalisation by evaluating at lengths beyond the training maximum rather than assuming the scheme extrapolates.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Learned absolute | Simple, flexible within the trained range | No representation past the table, extra parameters | Inputs exceed the trained length |
| Fixed sinusoids | No parameters, deterministic, smooth | Weak length extrapolation in practice | Long contexts are required |
| Relative bias (ALiBi-style) | Cheap, distance-aware, extrapolates better | No absolute position, bias design is hand-set | Tasks need exact index information |
| RoPE | Relative behaviour with absolute inputs, strong empirical record | Frequency base and scaling need tuning for long context | The serving kernel lacks the rotation implementation |
Good for
Simple, flexible within the trained range
Cost
No representation past the table, extra parameters
Fails when
Inputs exceed the trained length
- Inputs exceed the trained length
- Long contexts are required
- Tasks need exact index information
- The serving kernel lacks the rotation implementation
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Learned absolute is good for Simple, flexible within the trained range, costs No representation past the table, extra parameters, and fails when Inputs exceed the trained length. Fixed sinusoids is good for No parameters, deterministic, smooth, costs Weak length extrapolation in practice, and fails when Long contexts are required. Relative bias (ALiBi-style) is good for Cheap, distance-aware, extrapolates better, costs No absolute position, bias design is hand-set, and fails when Tasks need exact index information. RoPE is good for Relative behaviour with absolute inputs, strong empirical record, costs Frequency base and scaling need tuning for long context, and fails when The serving kernel lacks the rotation implementation
Failure modes
- Position information dropped at export. A traced or quantised graph loses the position addition and outputs become order-insensitive. Debug: shuffle two tokens and confirm the logits change.
- Encoding added after the first block. Position enters too late to affect the first attention layer. Debug: hook the input embedding output and check that position is present before block one.
- Extrapolation past the table. Indices beyond the trained range map to zero or an unknown row and quality collapses. Debug: evaluate loss at increasing lengths and find the point where it departs from the in-range curve.
In production
Monitor quality by input length bucket rather than in aggregate, because position schemes fail at the tail. When extending context, change the encoding scaling deliberately and re-evaluate on long inputs before raising the advertised limit.
Common misconceptions
People think: Sinusoidal encodings extrapolate to any length.
Actually: The formula is defined for any index, but the model has only learned to use the range it saw; performance usually degrades beyond it.
People think: Positional encoding is a minor detail compared with attention.
Actually: It determines whether the model can use order at all and how far its context generalises, and changing it invalidates long-context behaviour.
Interview angle
A strong answer starts from permutation invariance, then contrasts absolute, relative and rotary schemes on what they encode and how they extrapolate. It should name the failure at lengths beyond training. Weak answers describe sine waves without explaining why order is missing in the first place or what changes with RoPE.
Sources
- Vaswani et al., Attention Is All You Need — sinusoidal positional encodings.
- Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding — rotary position encoding and relative behaviour.
- Press, Smith & Lewis, Train Short, Test Long: ALiBi — linear distance biases and length extrapolation.