Distillation and synthetic data

The two techniques behind “we replaced a frontier model with something 20x cheaper”, and they are usually the same project: the big model generates the data, the small model learns from it.

frontier model ──▶ outputs on your traffic
                        │
                   filter + verify
                        │
                        ▼
                 small model, fine-tuned

Distillation

Train a small student to reproduce a large teacher’s behaviour on your task. Not on everything the teacher can do — on the narrow slice you actually serve, which is why it works.

Response distillation is what you will do. Logit distillation needs access to the teacher’s raw probabilities, which closed APIs do not expose, so it is limited to open-weight teachers.

Rationale distillation — training on the teacher’s chain of thought as well as its answer — reliably beats response-only on reasoning tasks, and it is a cheap upgrade since you are already paying for the generation.

When it pays

The economics only work at volume: you pay the teacher once per training example and save on every inference afterwards. For a low-traffic endpoint the fine-tuning effort never amortises.

Realistic outcome on a narrow task: a small model within a point or two of the frontier at a fraction of the cost and latency. On a broad task it will disappoint, and the mistake is expecting generality you never trained for.

Gotcha: check the teacher’s terms of service. Several providers prohibit using their outputs to train a competing model. This is a legal question before it is a technical one, and mentioning it unprompted reads as senior.

Synthetic data

The reason distillation is possible, and useful on its own when real data is scarce, sensitive or imbalanced.

Uses that hold up:

• Bootstrapping. No labelled data yet, and you need a starting set.
• Covering the tail. Real traffic has 40 examples of the rare case; you generate 400.
• Privacy. Synthetic records instead of production PII in a dev environment or an eval set.
• Evaluation sets, with care — see the warning below.

Generation needs seeds and structure

Asking a model for “1,000 customer support questions” gives you 1,000 variations of about six questions. Diversity has to be engineered:

for topic in topics:
    for persona in personas:
        for style in ("terse", "rambling", "angry"):
            yield generate(topic, persona, style)

Seed from real artefacts where you can — actual product names, actual document structures, actual error messages — because the failure mode is synthetic data that is fluent, plausible and unlike your traffic.

Filtering is the whole quality story

Generated data contains the teacher’s mistakes, and training on them teaches them. The pipeline that works:

1. Generate more than you need, at a higher temperature for diversity.
2. Verify — execute the code, check the schema, confirm the answer against a source of truth where one exists.
3. Filter — drop near-duplicates, drop anything a judge scores poorly.
4. Balance — do not let one topic dominate the mix.

Verification is what separates this from noise. Where an answer can be checked mechanically — code that runs, JSON that validates, arithmetic — the yield is high. Where it cannot, you are propagating the teacher’s errors with extra steps.

Model collapse

Training repeatedly on synthetic output degrades the distribution: rare cases disappear, and the model converges toward its own average. Keep real data in the mix, and treat a fully synthetic pipeline over several generations as a known failure mode rather than a scaling strategy.

The trap: synthetic eval sets

Gotcha: an eval set generated by the same model family you are testing will flatter it. The questions arrive in the phrasing the model finds easy and skip the ambiguity real users produce.

Synthetic evals are acceptable as a regression harness — catching a change that breaks something — and not as evidence of real-world quality. That needs real queries, even a hundred of them. See Building eval sets.

Where this sits in the escalation

Prompt → few-shot → RAG → tools → distil what you now know works. Distilling is a cost optimisation applied to a working system, not a way to build one. You need the teacher performing well first, because you are copying its behaviour — including its flaws.

Related

• Supervised fine-tuning and instruction tuning — the fine-tuning step underneath
• Quantization — the other way to make a model cheaper
• Fine-Tuning vs RAG vs Prompt Engineering — which problem you are solving
• Which model for which task — routing to the small model once you have it