Skip to content
AI Engineer Guide.
juniorReportedcurated

Why did transformers replace RNNs and LSTMs for language modeling?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What inference cost do transformers pay that RNNs avoid?
  • Decode reads the KV cache for the whole prefix on every step, so per-token latency grows with context and concurrency multiplies the memory.
  • Attention itself costs O(n²) in prefill, while a recurrent model's per-step cost is constant in sequence length.
  • That is why long-context serving is a cache-and-bandwidth problem, and why GQA, quantisation and paging exist at all.
  • The transformer still wins because training and prefill parallelise across the whole sequence and decode batches across requests; a recurrence cannot parallelise within a sequence at all, so it never uses the device the same way.
Why does causal masking make training sample-efficient per forward pass?
  • An RNN has to step through the sequence to produce the same targets, so n positions cost n sequential passes instead of one parallel pass.
  • Without the mask each position could copy its target, so those n targets would be worthless and the model would learn nothing transferable.
  • The result is that compute per gradient step is spent on the whole sequence at once instead of one prefix at a time.

Red flags

  • Claims transformers are strictly cheaper at inference
  • Cannot explain why recurrence blocks parallelism

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals