seniorReportedcurated
Why is RoPE called a relative position scheme even though each token is rotated by its absolute index, and how do you extend its context?
30-second answer
- RoPE rotates each query and key by an angle proportional to its absolute position, so position enters as a rotation rather than as an added vector.
- Because rotations compose, the score ⟨R_m q, R_n k⟩ equals ⟨q, R_(n−m) k⟩, which depends only on the offset n − m.
- That is why it behaves as a relative scheme while using absolute indices, and why it needs no learned position parameters and generalises better than a learned absolute table.
- Extending context means rescaling the rotation base or frequencies, by position interpolation, NTK scaling or YaRN, usually with a short fine-tune afterwards.
- The trap is validating with perplexity: naive extension can keep loss healthy while long-range retrieval collapses, so use needle-in-a-haystack and task evals.
Likely follow-ups
Open a question to rehearse the answer.
Why does the relative-offset property not guarantee extrapolation?
- The score depends only on n − m, but the model has only learned to use offsets inside the range it trained on, so a longer sequence asks it to apply that pattern outside the range.
- Relative form is necessary but not sufficient, because the encoding can express any offset while the learned weights still degrade past the trained length.
- That is why extension changes the frequency base or scaling deliberately and re-evaluates at lengths beyond the training maximum before raising the advertised limit.
Why can perplexity look fine while long-context recall degrades?
- Perplexity averages over every token, and most tokens in a long document are predictable from local context that the model still handles well.
- Retrieving one fact at one distant offset is a rarer and harder event, so a small loss change can hide a large drop in it.
- Naive frequency scaling can leave the model fluent while its attention loses the ability to resolve distant positions.
- That is why extension is validated with needle-in-a-haystack and task-level evals rather than with training loss.
Red flags
- Says RoPE adds a position vector to the embedding
- Claims RoPE has learned position parameters
Concepts this tests
Tags
Source: github.com