Track
Deep learning & transformers
Attention, tokenization, inference mechanics, and the transformer block.
- Neural networksconcept · T1
- Weights, biases, activationsconcept · T1
- MLPconcept · T1
- Forward passconcept · T1
- Backpropagationconcept · T2
- Tensorsconcept · T1
- Hardwareconcept · T1
- Training loopconcept · T2
- Tokenisationconcept · T1
- Tokenizer algorithmsconcept · T1
- Vocabularyconcept · T1
- Subword tokenisationconcept · T1
- Language modellingconcept · T1
- Transformerconcept · T1
- Encoder/decoderconcept · T1
- Self-attentionconcept · T1
- Q/K/Vconcept · T1
- Scaled dot-product attentionconcept · T1
- Multi-head attentionconcept · T1
- Cross-attentionconcept · T2
- Causal maskingconcept · T1
- Positional encodingconcept · T1
- Feed-forward blockconcept · T1
- Residuals & layer normconcept · T1
- Pretraining objectivesconcept · T2
- Context windowconcept · T1
- KV cacheconcept · T1
- Prefill & decodeconcept · T1
- GQA & MQAconcept · T2
- RMSNormconcept · T2
- MoEconcept · T2
- Long-context attentionconcept · T2
- Scaling lawsconcept · T2