Maths & CS fundamentals
14 questions
- Why did transformers replace RNNs and LSTMs for language modeling?junior
- Walk me through what happens inside a single transformer decoder block.junior
- Explain self-attention step by step. What exactly are Q, K, and V, and why are K and V separate?junior
- Why does scaled dot-product attention divide by √d_k?mid
- What is causal masking, why is it needed, and why apply it before the softmax?junior
- Why use multiple attention heads instead of one head of the full model width?mid
- Pre-norm versus post-norm: what changes, and why did essentially everyone move to pre-norm and RMSNorm?senior
- What does the feed-forward network do, where do most parameters live, and why is SwiGLU the default?mid
- Why is RoPE called a relative position scheme even though each token is rotated by its absolute index, and how do you extend its context?senior
- What is FlashAttention, why is it exact rather than approximate, and what does it not fix?senior
- Walk me through BPE training and encoding. Why start from bytes, and what does vocabulary size trade off?mid
- Why do LLMs use subword tokenization rather than words or characters, and what does that choice cost you?junior
- If your vocabulary is 100K tokens, how do you reason about whether that is the right size, and what does it cost?senior
- What does perplexity measure, and why can't you compare perplexity across models with different tokenizers?senior