Deep learning
Enough to reason about transformers, since that’s where it leads. Every structural trick here — residuals, normalisation, activations — reappears in an LLM.
| # | File | The question it answers |
|---|---|---|
| 01 | Neural network basics | why depth needs non-linearity; the training loop’s four silent bugs |
| 02 | Activation functions | why ReLU beat sigmoid; GELU and SwiGLU in modern LLMs |
| 03 | Training deep networks | NaN loss debugging, warmup, fp16 vs bf16, fitting a model in memory |
| 04 | Convolutional networks | inductive bias, residuals, CNN vs Vision Transformer |
| 05 | RNNs and LSTMs | why they lost to transformers — which is the interview value |
| 06 | Normalisation and residual connections | BatchNorm vs LayerNorm, RMSNorm, pre-norm vs post-norm |
| 07 | Transfer learning | feature extraction, LoRA/PEFT, and fine-tune vs retrieve |
The three answers worth having ready
Why residual connections matter. They give the gradient an identity path, and reduce each block to learning a difference from identity. They fixed the degradation problem where deeper networks had higher training error — an optimisation failure, not overfitting.
Why LayerNorm in transformers. Variable sequence lengths, small per-device batches, and one-token-at-a-time generation all make batch statistics unusable. LayerNorm depends only on the sample, so it behaves identically at any batch size and in both training and inference.
Fine-tuning teaches form, not facts. Knowledge baked into weights can’t be updated or cited. If the requirement is “answer from our documents”, that’s retrieval. Try prompt, then few-shot, then RAG, then fine-tuning.