AI & ML / Deep learning / 09_generative_models.md

Deep generative models

Updated 6 interview angles 5 min read source
On this page8
  1. The four families
  2. VAEs: a latent space you can sample
  3. GANs: no likelihood, a critic instead
  4. Diffusion: learn to denoise
  5. Normalizing flows
  6. Choosing
  7. Related
  8. Interview angle

Deep generative models

The CS236 territory. Every model here answers the same question — how do you learn a distribution you can sample from — and they differ in whether they model the density explicitly, and what they give up to make it tractable.

The four families

Family Learns Sampling Likelihood
Autoregressive p(x) = Π p(xᵢ | x<ᵢ) slow, sequential exact
VAE a latent space + decoder fast, one pass lower bound
GAN a generator, adversarially fast, one pass none
Diffusion how to denoise, step by step slow, iterative bound

Autoregressive is the one that won, and it is worth saying why: an LLM is exactly this — factorise the joint into a product of conditionals and predict the next token. Exact likelihood, trivially stable training, and the cost is that generation is inherently sequential. Everything about inference latency in Inference and serving follows from that.

VAEs: a latent space you can sample

An encoder maps x to a distribution over latents, a decoder maps back, and the loss has two terms:

python 
# Reconstruction: does the decoder rebuild the input?
recon = F.mse_loss(decoder(z), x, reduction="sum")

# KL: is the latent distribution close to the prior N(0, I)?
kl = -0.5 * (1 + logvar - mu.pow(2) - logvar.exp()).sum()

loss = recon + beta * kl

The KL term is what makes the latent space continuous and samplable — you can draw z ~ N(0, I) and decode it into something plausible, which a plain autoencoder cannot do because its latent space has holes.

The reparameterisation trick is the mechanism that makes it trainable:

python 
# gradient flows through mu and std
z = mu + torch.randn_like(std) * std

Sampling is not differentiable; sampling noise and scaling it is. That one line is what lets backprop reach the encoder.

Gotcha: VAE samples are blurry, and the reason is the objective. An MSE reconstruction loss is maximised by predicting the conditional mean, and the mean of several plausible images is a blur. It is not a bug to tune away.

GANs: no likelihood, a critic instead

A generator makes samples, a discriminator judges real from fake, and they train against each other. The result is sharp — the discriminator punishes blur, which is exactly what the VAE objective rewards.

The cost is that there is no likelihood and no reliable convergence signal. The loss going down means little; the two networks are chasing each other. Named failure modes:

  • Mode collapse — the generator finds one output that fools the discriminator and produces only that.
  • Non-convergence — the pair oscillates instead of settling.
  • Vanishing discriminator gradient — a discriminator that gets too good stops giving the generator anything to learn from.

Mitigations worth naming: Wasserstein loss with gradient penalty, spectral normalisation, two-timescale learning rates. As of 2026 GANs have largely lost image generation to diffusion, and remain useful where sampling must be a single fast forward pass.

Diffusion: learn to denoise

Add Gaussian noise to data over many steps until it is pure noise, then train a network to reverse one step. Generation runs the reverse chain from noise.

python 
# Training is startlingly simple: predict the noise you added.
t = torch.randint(0, T, (batch,))
noise = torch.randn_like(x0)
# closed form, any t
xt = sqrt_acp[t] * x0 + sqrt_1macp[t] * noise
loss = F.mse_loss(model(xt, t), noise)

Two properties explain why diffusion took over. Training is stable — it is a regression on noise, with no adversarial game — and the closed form for xt means you can jump to any timestep without simulating the chain.

The cost is sampling: the reverse chain is many forward passes. That is what DDIM, distillation and consistency models attack, and it is the same latency-versus-quality trade as everywhere else.

Classifier-free guidance is the conditioning mechanism to know: train with the condition randomly dropped, then at sampling time extrapolate away from the unconditional prediction. Higher guidance means more prompt adherence and less diversity — the knob behind every text-to-image guidance_scale.

Normalizing flows

An invertible network with a tractable Jacobian, so the change-of-variables formula gives exact likelihood, and the inverse gives sampling. Elegant, and the architectural constraint — every layer invertible with a cheap determinant — costs enough expressiveness that they are now niche outside density estimation and some scientific work.

Choosing

Need Reach for
Text, code, sequences autoregressive
Images, audio, video diffusion
One-pass sampling, low latency GAN or a distilled diffusion model
A structured latent space VAE
Exact density flow or autoregressive

Interview angle 6

  • “What are the families of generative model?” - autoregressive (exact likelihood, sequential sampling), VAEs (latent space, likelihood bound), GANs (no likelihood, adversarial), diffusion (iterative denoising). They differ in whether the density is explicit and what they trade for tractability.
  • “Why did autoregressive models win for text?” - factorising into next-token conditionals gives exact likelihood and completely stable training. The price is that generation is inherently sequential, which is the source of every LLM latency problem.
  • “Why are VAE samples blurry?” - the objective. An MSE reconstruction loss is minimised by predicting the conditional mean, and the mean of several plausible outputs is a blur. GANs are sharp because a discriminator punishes exactly that.
  • “What is the reparameterisation trick and why is it needed?” - sampling is not differentiable, so gradients cannot reach the encoder. Writing z = mu + eps * std with eps drawn separately moves the randomness outside the path, and backprop flows through mu and std.
  • “Why did diffusion beat GANs for images?” - training stability. Diffusion is a regression that predicts the noise it added, with a closed form for any timestep; GANs are a two-player game with mode collapse and no reliable convergence signal. Diffusion pays for it at sampling time.
  • “What is classifier-free guidance?” - train with the condition randomly dropped so the model learns both conditional and unconditional prediction, then at sampling extrapolate away from the unconditional one. It is the guidance_scale knob: more prompt adherence, less diversity.
Previous
Backpropagation