Skip to content
AI Engineer Guide.

Track

Deep learning & transformers

Attention, tokenization, inference mechanics, and the transformer block.

  1. Neural networksconcept · T1
  2. Weights, biases, activationsconcept · T1
  3. MLPconcept · T1
  4. Forward passconcept · T1
  5. Backpropagationconcept · T2
  6. Tensorsconcept · T1
  7. Hardwareconcept · T1
  8. Training loopconcept · T2
  9. Tokenisationconcept · T1
  10. Tokenizer algorithmsconcept · T1
  11. Vocabularyconcept · T1
  12. Subword tokenisationconcept · T1
  13. Language modellingconcept · T1
  14. Transformerconcept · T1
  15. Encoder/decoderconcept · T1
  16. Self-attentionconcept · T1
  17. Q/K/Vconcept · T1
  18. Scaled dot-product attentionconcept · T1
  19. Multi-head attentionconcept · T1
  20. Cross-attentionconcept · T2
  21. Causal maskingconcept · T1
  22. Positional encodingconcept · T1
  23. Feed-forward blockconcept · T1
  24. Residuals & layer normconcept · T1
  25. Pretraining objectivesconcept · T2
  26. Context windowconcept · T1
  27. KV cacheconcept · T1
  28. Prefill & decodeconcept · T1
  29. GQA & MQAconcept · T2
  30. RMSNormconcept · T2
  31. MoEconcept · T2
  32. Long-context attentionconcept · T2
  33. Scaling lawsconcept · T2