Multi-task and meta-learning

The CS330 territory, and the framing that makes it interview-useful: transfer learning moves knowledge between two tasks; multi-task learning shares one model across many; meta-learning learns how to learn a new one quickly. Different problems, routinely conflated.

Multi-task learning

One model, several objectives, a shared trunk:

class MultiTask(nn.Module):
    def __init__(self, encoder, dims):
        super().__init__()
        # shared
        self.encoder = encoder
        self.heads = nn.ModuleDict(
            {name: nn.Linear(768, d) for name, d in dims.items()}
        )

    def forward(self, x, task: str):
        return self.heads[task](self.encoder(x))

The shared encoder is a regulariser: each task’s data constrains the representation, so tasks with little data borrow structure from tasks with a lot. That is the upside, and it is real when the tasks are related.

Negative transfer is the downside, and it is the thing to name. Unrelated tasks fight over capacity and every task ends up worse than its own dedicated model. The diagnosis is per-task metrics against single-task baselines — a falling average hides one task collapsing while another improves.

The loss is a weighted sum, and the weights matter more than people expect:

loss = sum(w[t] * losses[t] for t in tasks)

Tasks with larger-magnitude losses or steeper gradients dominate. Fixed weights tuned by hand is the honest baseline; uncertainty weighting and GradNorm are the named alternatives that learn them.

Gotcha: in production, multi-task usually loses to separate models for a non-technical reason — deployment coupling. One model means one release cadence for every task, and a regression in one blocks all of them. Say that; it is the consideration a research answer misses.

Meta-learning: learn to learn

The setup is different. Instead of samples you have tasks, each with a small support set (to adapt on) and a query set (to evaluate on), and the goal is a model that adapts to an unseen task from a handful of examples.

MAML is the one to be able to describe. Learn an initialisation from which one or two gradient steps solve a new task:

for task in batch_of_tasks:
    fast = clone(model)
    # inner loop: adapt on the support set
    for _ in range(steps):
        loss = criterion(fast(task.support_x), task.support_y)
        fast = sgd_step(fast, loss, lr=inner_lr)
    # outer loop: how good was the ADAPTED model on the query set?
    meta_loss += criterion(fast(task.query_x), task.query_y)

# gradient through the inner steps
meta_loss.backward()
meta_optimizer.step()

The idea in one sentence: the outer loss is measured after adaptation, so the initialisation is optimised for adaptability rather than for immediate performance.

That backward pass goes through the inner updates, which means second-order gradients — expensive and memory-hungry. First-order MAML and Reptile drop that term and work nearly as well, which is why they are what people actually run.

Prototypical networks are the pragmatic one: no inner loop at all, just a good embedding and a nearest-centroid rule. Often competitive, and far simpler.

What replaced it in practice

This is the honest 2026 framing, and it is what an interviewer is listening for. In-context learning did to few-shot what pretraining did to feature engineering. A frontier model given five examples in the prompt solves many few-shot problems with no gradient step, no episode construction and no meta-training.

So the field’s centre moved: meta-learning remains relevant where in-context learning cannot reach — small on-device models, non-language modalities, genuine per-user personalisation, robotics — and the vocabulary is still worth having because the framing survived. Prompting with examples is few-shot learning; the model was meta-trained by pretraining.

Related

• Transfer learning — the two-task case
• Fine-Tuning vs RAG vs Prompt Engineering — cheap per-task adaptation that actually ships
• Prompt Engineering — in-context learning
• ML system design — one model or many, as a system decision