Skip to content
AI Engineer Guide.
seniorReportedcurated

Why is RoPE called a relative position scheme even though each token is rotated by its absolute index, and how do you extend its context?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why does the relative-offset property not guarantee extrapolation?
  • The score depends only on n − m, but the model has only learned to use offsets inside the range it trained on, so a longer sequence asks it to apply that pattern outside the range.
  • Relative form is necessary but not sufficient, because the encoding can express any offset while the learned weights still degrade past the trained length.
  • That is why extension changes the frequency base or scaling deliberately and re-evaluates at lengths beyond the training maximum before raising the advertised limit.
Why can perplexity look fine while long-context recall degrades?
  • Perplexity averages over every token, and most tokens in a long document are predictable from local context that the model still handles well.
  • Retrieving one fact at one distant offset is a rarer and harder event, so a small loss change can hide a large drop in it.
  • Naive frequency scaling can leave the model fluent while its attention loses the ability to resolve distant positions.
  • That is why extension is validated with needle-in-a-haystack and task-level evals rather than with training loss.

Red flags

  • Says RoPE adds a position vector to the embedding
  • Claims RoPE has learned position parameters

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals