Why it matters
The transformer is the architecture under every model you call. That matters less for writing one from scratch than for predicting behaviour: why long prompts are expensive before the first token appears, why output length dominates latency, why a context extension changes quality, why attention memory grows quadratically while parameters grow linearly with depth.
It also gives you the vocabulary to read papers and model cards. "32 layers, hidden size 4096, 32 heads, grouped-query attention with 8 KV heads" is a precise statement about compute, memory and serving cost once you know how the blocks fit together.
Mental model: a shared whiteboard, refined in rounds
Picture a whiteboard with one column per token position. Each round, every column reads from all the other columns (attention), writes a summary into its own slot, then privately rewrites that summary through a small network (feed-forward). A residual connection keeps the previous board state, and the next round adds to it rather than replacing it.
The whiteboard analogy captures two things people often miss: information moves between positions only in the attention step, and every block adds to a persistent stream rather than passing a baton.
It breaks in one place worth naming. The rounds are not a pipeline where layer 3 finishes before layer 4 starts conceptually — they are a deep composition, and the model has no explicit intermediate outputs. Interpretability work tries to read the board, but the model does not expose a clean "this layer decided X".
How it works
Step 1 of 6
Ids become vectors
Each token id indexes the embedding matrix, giving one vector per position. The matrix is learned, and its size scales with vocabulary, not with context length.
Active layers
- Token ids
- Embedding + position
Hover a layer to read what it holds.
Next: Position enters
- Mixing
- attention only
- Depth
- same block, L times
- Parameters
- ≈12d²L + Vd
Figure. A decoder-only transformer from token ids to a next-token distribution. The attention sublayer is the only band where information moves between positions; every block adds a delta to the same residual stream, and the dash-dot loop is depth repeating that composition. Head count, layer count and the positional scheme are omitted; widths are illustrative.
What to notice. The first residual lane leaves the embedding band and lands after attention; the second leaves that add and lands after the feed-forward block, so a block cannot replace the stream, only add to it. Each sublayer band contains its own pre-norm, which is why the residual edges are never rescaled. Parameter count follows depth rather than context length: the same block repeated L times, with the vocabulary in the embedding and unembedding matrices. The causal mask sits inside the attention band, not in a layer of its own.
Text version
A vertical stack. Token ids become embedding vectors with position added, then enter a block group repeated L times. Inside the block, a normalised causal attention sublayer mixes positions and its output is added to the residual stream; a normalised feed-forward sublayer transforms each position and its output is added to the stream again. A residual lane runs from the embedding band past attention into the first add, and a second runs from that add past the feed-forward block into the second add; a dash-dot lane labelled next block returns from the second add to the block group. A final norm and unembedding produce a next-token distribution. Six steps cover embeddings, position, attention mixing, the attention residual, the per-position feed-forward, and depth with readout.
- Tokenise and embed. Token ids index an embedding matrix, producing one vector per position. The embedding matrix is learned.
- Inject position. Attention alone is permutation-invariant, so positional information is added or rotated in. Without it, word order would be invisible.
- Enter the block stack. Each block applies pre-norm, multi-head self-attention, a residual add, pre-norm, a feed-forward network, and another residual add. Blocks are identical in shape, not in weights.
- Mix across positions. Attention computes, for each position, a weighted average of value vectors, with weights from query–key similarity. In a decoder, a causal mask prevents a position from reading the future.
- Transform per position. The feed-forward network is a two-layer MLP applied independently at each position, typically expanding to four times the model width and projecting back. It holds most of the parameters.
- Produce logits. A final norm, then an unembedding matrix maps each position's vector to vocabulary scores; softmax turns them into a next-token distribution.
- Train with next-token prediction. For each position, predict the following token and compute cross-entropy against the true token. The causal mask lets one forward pass supply a training signal at every position.
- Serve in two phases. Prefill processes the whole prompt in parallel; decode produces one token per step and reuses cached keys and values.
- Per layer
- attention + feed-forward, with residuals
- Parameters
- ≈ 12d² per layer, plus embeddings
- Attention cost
- grows quadratically with sequence length
- Decode
- one token per forward pass
The math
Attention is a soft, differentiable lookup; the details live in self-attention, but the shape of the cost matters here:
Parameter counting is the other useful piece of arithmetic. For a width- model with a feed-forward expansion of , each block holds roughly attention parameters and feed-forward parameters:
where is the number of layers and the vocabulary size. Context length does not appear: longer prompts cost compute and cache memory, not parameters.
A minimal block
import torch
import torch.nn as nn
class Block(nn.Module):
def __init__(self, d: int, heads: int):
super().__init__()
self.heads, self.d_head = heads, d // heads
self.norm1, self.norm2 = nn.LayerNorm(d), nn.LayerNorm(d)
self.qkv = nn.Linear(d, 3 * d, bias=False)
self.out = nn.Linear(d, d, bias=False)
self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
def forward(self, x: torch.Tensor) -> torch.Tensor:
b, n, d = x.shape # batch, sequence, width
q, k, v = self.qkv(self.norm1(x)).split(d, dim=-1)
shape = (b, n, self.heads, self.d_head)
q, k, v = (t.view(shape).transpose(1, 2) for t in (q, k, v))
scores = q @ k.transpose(-2, -1) / self.d_head ** 0.5
mask = torch.triu(torch.ones(n, n, dtype=torch.bool, device=x.device), diagonal=1)
scores = scores.masked_fill(mask, float("-inf")) # causal: no future tokens
attn = scores.softmax(dim=-1) @ v
attn = attn.transpose(1, 2).reshape(b, n, d)
x = x + self.out(attn) # residual after attention
return x + self.ffn(self.norm2(x)) # residual after feed-forward
block = Block(d=64, heads=4)
print(block(torch.randn(2, 6, 64)).shape) # torch.Size([2, 6, 64])The two places this goes wrong in practice are the mask and the reshape. Transposing before the reshape in the wrong order silently shuffles head outputs, and a mask with the wrong shape broadcasts into a no-op.
Trade-offs
| Architecture | Good for | Cost | Fails when |
|---|---|---|---|
| Encoder-only | Classification, embeddings, retrieval | Bidirectional attention over the full input | You need to generate text |
| Decoder-only | Generation, chat, in-context learning | Sequential decode, KV cache memory | The task needs a compact fixed embedding for a long document |
| Encoder-decoder | Translation, summarisation with a fixed source | Two stacks, cross-attention | Latency budget is tight and the task is a simple continuation |
| Mixture-of-experts | Large parameter counts at lower compute per token | Routing complexity, memory for all experts | Traffic is small or batch sizes are tiny |
Good for
Classification, embeddings, retrieval
Cost
Bidirectional attention over the full input
Fails when
You need to generate text
- You need to generate text
- The task needs a compact fixed embedding for a long document
- Latency budget is tight and the task is a simple continuation
- Traffic is small or batch sizes are tiny
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Encoder-only is good for Classification, embeddings, retrieval, costs Bidirectional attention over the full input, and fails when You need to generate text. Decoder-only is good for Generation, chat, in-context learning, costs Sequential decode, KV cache memory, and fails when The task needs a compact fixed embedding for a long document. Encoder-decoder is good for Translation, summarisation with a fixed source, costs Two stacks, cross-attention, and fails when Latency budget is tight and the task is a simple continuation. Mixture-of-experts is good for Large parameter counts at lower compute per token, costs Routing complexity, memory for all experts, and fails when Traffic is small or batch sizes are tiny
Failure modes
Future-token leakage. A mask is missing, mis-shaped, or applied after the softmax, and training loss looks excellent because the model can see the answer. Debugging step: run a forward pass, perturb a token at position , and assert the logits at position do not change.
Position and cache drift. During incremental decoding, positions or cache offsets are off by one, so the model conditions on the wrong context. Debugging step: compare a cached generation against a full recomputation for the same prefix; the first divergence tells you which step is misaligned.
Attention logits overflow in low precision. Large scores saturate float16 softmax and produce nan or uniform attention. Debugging step: log the maximum absolute attention logit per layer, and keep the softmax in float32 if needed.
Context beyond the trained range. The model accepts more tokens than its positional scheme was trained for and quality degrades without an error. Debugging step: evaluate accuracy at increasing lengths on your own task rather than trusting the advertised limit.
In production
Separate the two phases in your mental accounting. Prefill cost scales with prompt tokens and is compute-bound; decode cost scales with output tokens and is memory-bandwidth-bound because every step reads the whole cache. That is why time-to-first-token and tokens-per-second move independently, and why batching helps decode far more than it helps a single long prefill.
Memory is parameters plus cache. Weights dominate at short context; the cache dominates at long context and high concurrency. Quantising weights reduces the first term and does nothing for the second — see KV cache for that arithmetic.
Common misconceptions
People think: Attention is where most of a transformer's knowledge lives.
Actually: Attention mixes information between positions, but the feed-forward blocks hold roughly two-thirds of each layer's parameters and much of the model's stored associations.
People think: A transformer can attend to anything in its context, so position does not matter much.
Actually: Position is injected explicitly, and models attend unevenly across long contexts — accuracy often degrades for evidence in the middle of a long prompt.
A third: that the architecture alone explains capability. Scale, data, training objective and post-training decide what the model can do; the transformer is the substrate that makes those tractable.
Interview angle
Architecture questions usually start at "walk me through a forward pass" and end at cost. Be ready to count parameters per layer, explain why prefill is parallel and decode is not, and describe what a causal mask protects against. Strong answers connect each component to a serving consequence. Pair this with self-attention and prefill and decode, and see the model questions at /interview.
Sources
- Vaswani et al., Attention Is All You Need — the original architecture and its complexity arguments: https://arxiv.org/abs/1706.03762
- Harvard NLP, The Annotated Transformer — line-by-line implementation of the paper: https://nlp.seas.harvard.edu/annotated-transformer/
- Jay Alammar, The Illustrated Transformer — the clearest visual walkthrough of blocks and attention: https://jalammar.github.io/illustrated-transformer/
- Lilian Weng, The Transformer Family v2 — variants, positional schemes and efficiency changes since the original: https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/