juniorReportedcurated
Why did transformers replace RNNs and LSTMs for language modeling?
30-second answer
- An RNN's hidden state depends on the previous step, so training cannot parallelise across the sequence and signal from early tokens must survive hundreds of recurrent steps.
- Self-attention compares every pair of positions in one matrix multiply, so any two tokens interact through a path of length one and the whole sequence is processed in parallel.
- With a causal mask, one forward pass produces a next-token loss at every position, which multiplies the training signal per unit of compute.
- The inference trade is real: an RNN keeps O(1) state per token, while a transformer pays O(n²) attention compute and an O(n) KV cache that grows with context and concurrency.
- Transformers are not strictly cheaper to serve; they are cheaper to train at scale, and the cache is the bill that arrives at inference.
Likely follow-ups
Open a question to rehearse the answer.
What inference cost do transformers pay that RNNs avoid?
- Decode reads the KV cache for the whole prefix on every step, so per-token latency grows with context and concurrency multiplies the memory.
- Attention itself costs O(n²) in prefill, while a recurrent model's per-step cost is constant in sequence length.
- That is why long-context serving is a cache-and-bandwidth problem, and why GQA, quantisation and paging exist at all.
- The transformer still wins because training and prefill parallelise across the whole sequence and decode batches across requests; a recurrence cannot parallelise within a sequence at all, so it never uses the device the same way.
Why does causal masking make training sample-efficient per forward pass?
- An RNN has to step through the sequence to produce the same targets, so n positions cost n sequential passes instead of one parallel pass.
- Without the mask each position could copy its target, so those n targets would be worthless and the model would learn nothing transferable.
- The result is that compute per gradient step is spent on the whole sequence at once instead of one prefix at a time.
Red flags
- Claims transformers are strictly cheaper at inference
- Cannot explain why recurrence blocks parallelism
Concepts this tests
Tags
Source: github.com