Transformers and LLMs
Rewritten for 2026. The 2017 paper is not what runs today — every frontier model is decoder-only with RoPE, SwiGLU, RMSNorm, pre-norm, GQA or MLA, and usually MoE.
Architecture
| # | File | Covers |
|---|---|---|
| 01 | Transformer architecture | the block, self-attention, why decoder-only won, what changed since 2017 |
| 02 | Attention variants | MHA → MQA → GQA → MLA, FlashAttention, attention sinks |
| 03 | Tokenization | BPE, token costs, why arithmetic fails, chat templates |
| 04 | Positional encoding | RoPE, and how context windows get extended |
Inference behaviour
| # | File | Covers |
|---|---|---|
| 05 | KV cache | prefill vs decode, cache sizing, PagedAttention, prefix caching |
| 06 | Mixture of Experts | total vs active parameters, load balancing, serving implications |
| 07 | Reasoning models and test-time compute | test-time compute, RLVR, GRPO, thinking budgets |
| 08 | Context windows | lost-in-the-middle, why advertised ≠ usable, long context vs RAG |
Using them
| # | File | Covers |
|---|---|---|
| 09 | Prompt Engineering | prompting techniques |
| 10 | Hallucinations, Function Calling, and Tool Use | why models confabulate, and grounding |
| 11 | Multimodal models | images as tokens, what vision models are bad at, injection via image |
| 12 | Programmatic prompt optimisation | a prompt is a parameter: DSPy, few-shot search, the eval-set prerequisite |
The four answers worth having ready
Why attention is quadratic, and what follows. Every pair of positions gets a score. That one fact drives context pricing, KV caching, FlashAttention, and the entire efficient-attention literature.
Prefill is compute-bound; decode is memory-bandwidth-bound. It explains why output tokens cost more than input tokens, why batching works, and why KV cache size — not FLOPs — caps concurrency.
Total vs active parameters. A 671B MoE with 37B active still needs memory for 671B. It is not “a 37B model”.
Test-time compute is a scaling axis. A smaller model with an adequate thinking budget can beat a much larger one at matched compute. That changed model selection.