Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 14 of 33

Transformer

0 of 33 complete

Why it matters

The transformer is the architecture under every model you call. That matters less for writing one from scratch than for predicting behaviour: why long prompts are expensive before the first token appears, why output length dominates latency, why a context extension changes quality, why attention memory grows quadratically while parameters grow linearly with depth.

It also gives you the vocabulary to read papers and model cards. "32 layers, hidden size 4096, 32 heads, grouped-query attention with 8 KV heads" is a precise statement about compute, memory and serving cost once you know how the blocks fit together.

Mental model: a shared whiteboard, refined in rounds

Picture a whiteboard with one column per token position. Each round, every column reads from all the other columns (attention), writes a summary into its own slot, then privately rewrites that summary through a small network (feed-forward). A residual connection keeps the previous board state, and the next round adds to it rather than replacing it.

The whiteboard analogy captures two things people often miss: information moves between positions only in the attention step, and every block adds to a persistent stream rather than passing a baton.

It breaks in one place worth naming. The rounds are not a pipeline where layer 3 finishes before layer 4 starts conceptually — they are a deep composition, and the model has no explicit intermediate outputs. Interpretability work tries to read the board, but the model does not expose a clean "this layer decided X".

How it works

One stream, L identical blocks, and where positions mixInteractive
signalresidualloop

Step 1 of 6

Ids become vectors

Each token id indexes the embedding matrix, giving one vector per position. The matrix is learned, and its size scales with vocabulary, not with context length.

Active layers

  • Token ids
  • Embedding + position

Hover a layer to read what it holds.

Next: Position enters

Mixing
attention only
Depth
same block, L times
Parameters
≈12d²L + Vd
Step 1/6 · Ids become vectors

Figure. A decoder-only transformer from token ids to a next-token distribution. The attention sublayer is the only band where information moves between positions; every block adds a delta to the same residual stream, and the dash-dot loop is depth repeating that composition. Head count, layer count and the positional scheme are omitted; widths are illustrative.

What to notice. The first residual lane leaves the embedding band and lands after attention; the second leaves that add and lands after the feed-forward block, so a block cannot replace the stream, only add to it. Each sublayer band contains its own pre-norm, which is why the residual edges are never rescaled. Parameter count follows depth rather than context length: the same block repeated L times, with the vocabulary in the embedding and unembedding matrices. The causal mask sits inside the attention band, not in a layer of its own.

Text version

A vertical stack. Token ids become embedding vectors with position added, then enter a block group repeated L times. Inside the block, a normalised causal attention sublayer mixes positions and its output is added to the residual stream; a normalised feed-forward sublayer transforms each position and its output is added to the stream again. A residual lane runs from the embedding band past attention into the first add, and a second runs from that add past the feed-forward block into the second add; a dash-dot lane labelled next block returns from the second add to the block group. A final norm and unembedding produce a next-token distribution. Six steps cover embeddings, position, attention mixing, the attention residual, the per-position feed-forward, and depth with readout.

  1. Tokenise and embed. Token ids index an embedding matrix, producing one vector per position. The embedding matrix is learned.
  2. Inject position. Attention alone is permutation-invariant, so positional information is added or rotated in. Without it, word order would be invisible.
  3. Enter the block stack. Each block applies pre-norm, multi-head self-attention, a residual add, pre-norm, a feed-forward network, and another residual add. Blocks are identical in shape, not in weights.
  4. Mix across positions. Attention computes, for each position, a weighted average of value vectors, with weights from query–key similarity. In a decoder, a causal mask prevents a position from reading the future.
  5. Transform per position. The feed-forward network is a two-layer MLP applied independently at each position, typically expanding to four times the model width and projecting back. It holds most of the parameters.
  6. Produce logits. A final norm, then an unembedding matrix maps each position's vector to vocabulary scores; softmax turns them into a next-token distribution.
  7. Train with next-token prediction. For each position, predict the following token and compute cross-entropy against the true token. The causal mask lets one forward pass supply a training signal at every position.
  8. Serve in two phases. Prefill processes the whole prompt in parallel; decode produces one token per step and reuses cached keys and values.
Per layer
attention + feed-forward, with residuals
Parameters
≈ 12d² per layer, plus embeddings
Attention cost
grows quadratically with sequence length
Decode
one token per forward pass

The math

Attention is a soft, differentiable lookup; the details live in self-attention, but the shape of the cost matters here:

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V,cost=O(n2d).\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V, \qquad \text{cost} = O(n^2 d).

Parameter counting is the other useful piece of arithmetic. For a width-dd model with a feed-forward expansion of 4d4d, each block holds roughly 4d24d^2 attention parameters and 8d28d^2 feed-forward parameters:

params≈12d2L+Vd,\text{params} \approx 12d^2 L + Vd,

where LL is the number of layers and VV the vocabulary size. Context length does not appear: longer prompts cost compute and cache memory, not parameters.

A minimal block

import torch
import torch.nn as nn
 
class Block(nn.Module):
    def __init__(self, d: int, heads: int):
        super().__init__()
        self.heads, self.d_head = heads, d // heads
        self.norm1, self.norm2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d, bias=False)
        self.out = nn.Linear(d, d, bias=False)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        b, n, d = x.shape                       # batch, sequence, width
        q, k, v = self.qkv(self.norm1(x)).split(d, dim=-1)
        shape = (b, n, self.heads, self.d_head)
        q, k, v = (t.view(shape).transpose(1, 2) for t in (q, k, v))
 
        scores = q @ k.transpose(-2, -1) / self.d_head ** 0.5
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool, device=x.device), diagonal=1)
        scores = scores.masked_fill(mask, float("-inf"))     # causal: no future tokens
        attn = scores.softmax(dim=-1) @ v
        attn = attn.transpose(1, 2).reshape(b, n, d)
 
        x = x + self.out(attn)                 # residual after attention
        return x + self.ffn(self.norm2(x))     # residual after feed-forward
 
block = Block(d=64, heads=4)
print(block(torch.randn(2, 6, 64)).shape)      # torch.Size([2, 6, 64])

The two places this goes wrong in practice are the mask and the reshape. Transposing before the reshape in the wrong order silently shuffles head outputs, and a mask with the wrong shape broadcasts into a no-op.

Trade-offs

ArchitectureGood forCostFails when
Encoder-onlyClassification, embeddings, retrievalBidirectional attention over the full inputYou need to generate text
Decoder-onlyGeneration, chat, in-context learningSequential decode, KV cache memoryThe task needs a compact fixed embedding for a long document
Encoder-decoderTranslation, summarisation with a fixed sourceTwo stacks, cross-attentionLatency budget is tight and the task is a simple continuation
Mixture-of-expertsLarge parameter counts at lower compute per tokenRouting complexity, memory for all expertsTraffic is small or batch sizes are tiny
Trade-off mapInteractive

Good for

Classification, embeddings, retrieval

Cost

Bidirectional attention over the full input

Fails when

You need to generate text

  1. You need to generate text
  2. The task needs a compact fixed embedding for a long document
  3. Latency budget is tight and the task is a simple continuation
  4. Traffic is small or batch sizes are tiny

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Encoder-only is good for Classification, embeddings, retrieval, costs Bidirectional attention over the full input, and fails when You need to generate text. Decoder-only is good for Generation, chat, in-context learning, costs Sequential decode, KV cache memory, and fails when The task needs a compact fixed embedding for a long document. Encoder-decoder is good for Translation, summarisation with a fixed source, costs Two stacks, cross-attention, and fails when Latency budget is tight and the task is a simple continuation. Mixture-of-experts is good for Large parameter counts at lower compute per token, costs Routing complexity, memory for all experts, and fails when Traffic is small or batch sizes are tiny

Failure modes

Future-token leakage. A mask is missing, mis-shaped, or applied after the softmax, and training loss looks excellent because the model can see the answer. Debugging step: run a forward pass, perturb a token at position t+1t+1, and assert the logits at position tt do not change.

Position and cache drift. During incremental decoding, positions or cache offsets are off by one, so the model conditions on the wrong context. Debugging step: compare a cached generation against a full recomputation for the same prefix; the first divergence tells you which step is misaligned.

Attention logits overflow in low precision. Large scores saturate float16 softmax and produce nan or uniform attention. Debugging step: log the maximum absolute attention logit per layer, and keep the softmax in float32 if needed.

Context beyond the trained range. The model accepts more tokens than its positional scheme was trained for and quality degrades without an error. Debugging step: evaluate accuracy at increasing lengths on your own task rather than trusting the advertised limit.

In production

Separate the two phases in your mental accounting. Prefill cost scales with prompt tokens and is compute-bound; decode cost scales with output tokens and is memory-bandwidth-bound because every step reads the whole cache. That is why time-to-first-token and tokens-per-second move independently, and why batching helps decode far more than it helps a single long prefill.

Memory is parameters plus cache. Weights dominate at short context; the cache dominates at long context and high concurrency. Quantising weights reduces the first term and does nothing for the second — see KV cache for that arithmetic.

Common misconceptions

People think: Attention is where most of a transformer's knowledge lives.

Actually: Attention mixes information between positions, but the feed-forward blocks hold roughly two-thirds of each layer's parameters and much of the model's stored associations.

People think: A transformer can attend to anything in its context, so position does not matter much.

Actually: Position is injected explicitly, and models attend unevenly across long contexts — accuracy often degrades for evidence in the middle of a long prompt.

A third: that the architecture alone explains capability. Scale, data, training objective and post-training decide what the model can do; the transformer is the substrate that makes those tractable.

Interview angle

Architecture questions usually start at "walk me through a forward pass" and end at cost. Be ready to count parameters per layer, explain why prefill is parallel and decode is not, and describe what a causal mask protects against. Strong answers connect each component to a serving consequence. Pair this with self-attention and prefill and decode, and see the model questions at /interview.

Sources