Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 22 of 33

Positional encoding

0 of 33 complete

Why it matters

Attention compares content, not order: the same tokens in a different sequence produce the same pairwise scores. Language, code and retrieval all depend on order, so the model needs a signal that distinguishes position one from position ten. The design question is whether to encode an absolute index, a relative distance, or a transformation of the vectors themselves, because that choice decides how the model behaves beyond its trained context length.

Mental model

Positional encoding is seat numbering in a theatre: the seats are identical, so without numbers nobody knows who is in front. The metaphor breaks because the numbers are not labels the model reads — they are added to or rotated into the content vectors, so position and meaning are entangled in the same space.

How it works

Sinusoidal table · five dimensions, eight positionsInteractive

hover or focus a cell

−1.000+1.000midpoint 0.000
position ↓dimension →
d₀d₁d₂d₃d₄
0
1
2
3
4
5
6
7

Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.

d_model
12
dimensions shown
d₀ – d₄
parameters
0 · fixed formula

Step 1 of 4

Position 0

Every sine is 0.000 and every cosine is 1.000 at position zero. The row is determined by the formula, not fitted, so the first position looks the same in every model that uses this scheme.

Next: Position 1

Step 1/4 · Position 0

Figure. The first five dimensions of a sinusoidal encoding for positions 0 to 7 at d_model = 12. Each row is one position's vector and each column is a sine or cosine of that position at a fixed frequency, so the table is a formula rather than a learned table and every entry can be recomputed. Shade is one signed scale from −1.000 to +1.000; the printed value carries the sign.

What to notice. Column d₀ is sin(pos) and completes a full cycle in these eight rows; d₂ divides by 10000^(2/12) ≈ 4.64 and turns more slowly; d₄ is the slowest shown and moves only from 0.000 to 0.319. The formula is defined at any index, which is what 'extrapolates in principle' means, but a model has only learned to use the range its training covered. The figure shows five of twelve dimensions and no learned or rotary scheme, so it cannot rank them.

Text version

An eight by five matrix. Rows are positions 0 to 7 and columns dimensions d₀ to d₄ of a 12-dimensional sinusoidal encoding. Row 0 prints 0.000, 1.000, 0.000, 1.000 and 0.000; row 1 prints 0.841, 0.540, 0.214, 0.977 and 0.046; row 2 prints 0.909, −0.416, 0.418, 0.909 and 0.093; row 3 prints 0.141, −0.990, 0.602, 0.798 and 0.139; row 4 prints −0.757, −0.654, 0.759, 0.651 and 0.185; row 5 prints −0.959, 0.284, 0.881, 0.474 and 0.230; row 6 prints −0.279, 0.960, 0.962, 0.275 and 0.275; row 7 prints 0.657, 0.754, 0.998, 0.063 and 0.319. Shade runs from teal at −1.000 through zero to accent at +1.000. Steps highlight position 0, position 1, the d₀ column across all eight positions, and the fast d₃ column against the slow d₄ column. Hovering a cell reports the position, the dimension and the printed value.

  1. Decide whether the model needs absolute position, relative distance, or both; the choice constrains the attention implementation.
  2. For fixed sinusoids, compute sine and cosine at geometrically spaced frequencies and add them to the input embeddings.
  3. For learned absolute encodings, allocate an embedding row per index and accept that indices beyond the table have no representation.
  4. For relative schemes, add a learned bias to attention scores based on the distance between query and key positions.
  5. For RoPE, rotate query and key vectors by an angle proportional to position, so their dot product depends on the offset.
  6. Validate length generalisation by evaluating at lengths beyond the training maximum rather than assuming the scheme extrapolates.

Trade-offs

ChoiceGood forCostFails when
Learned absoluteSimple, flexible within the trained rangeNo representation past the table, extra parametersInputs exceed the trained length
Fixed sinusoidsNo parameters, deterministic, smoothWeak length extrapolation in practiceLong contexts are required
Relative bias (ALiBi-style)Cheap, distance-aware, extrapolates betterNo absolute position, bias design is hand-setTasks need exact index information
RoPERelative behaviour with absolute inputs, strong empirical recordFrequency base and scaling need tuning for long contextThe serving kernel lacks the rotation implementation
Trade-off mapInteractive

Good for

Simple, flexible within the trained range

Cost

No representation past the table, extra parameters

Fails when

Inputs exceed the trained length

  1. Inputs exceed the trained length
  2. Long contexts are required
  3. Tasks need exact index information
  4. The serving kernel lacks the rotation implementation

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Learned absolute is good for Simple, flexible within the trained range, costs No representation past the table, extra parameters, and fails when Inputs exceed the trained length. Fixed sinusoids is good for No parameters, deterministic, smooth, costs Weak length extrapolation in practice, and fails when Long contexts are required. Relative bias (ALiBi-style) is good for Cheap, distance-aware, extrapolates better, costs No absolute position, bias design is hand-set, and fails when Tasks need exact index information. RoPE is good for Relative behaviour with absolute inputs, strong empirical record, costs Frequency base and scaling need tuning for long context, and fails when The serving kernel lacks the rotation implementation

Failure modes

  • Position information dropped at export. A traced or quantised graph loses the position addition and outputs become order-insensitive. Debug: shuffle two tokens and confirm the logits change.
  • Encoding added after the first block. Position enters too late to affect the first attention layer. Debug: hook the input embedding output and check that position is present before block one.
  • Extrapolation past the table. Indices beyond the trained range map to zero or an unknown row and quality collapses. Debug: evaluate loss at increasing lengths and find the point where it departs from the in-range curve.

In production

Monitor quality by input length bucket rather than in aggregate, because position schemes fail at the tail. When extending context, change the encoding scaling deliberately and re-evaluate on long inputs before raising the advertised limit.

Common misconceptions

People think: Sinusoidal encodings extrapolate to any length.

Actually: The formula is defined for any index, but the model has only learned to use the range it saw; performance usually degrades beyond it.

People think: Positional encoding is a minor detail compared with attention.

Actually: It determines whether the model can use order at all and how far its context generalises, and changing it invalidates long-context behaviour.

Interview angle

A strong answer starts from permutation invariance, then contrasts absolute, relative and rotary schemes on what they encode and how they extrapolate. It should name the failure at lengths beyond training. Weak answers describe sine waves without explaining why order is missing in the first place or what changes with RoPE.

Sources