Data poisoning, backdoors and model extraction
Prompt injection attacks a model at inference time. These attack it earlier or from further away — at training time, or by querying it until you have a copy. They matter for a Python engineer because the mitigations are mostly supply chain and API design, which is your job rather than the ML team’s.
The four attacks
| Attack | Attacker touches | Wants |
|---|---|---|
| Poisoning | training data | degraded behaviour |
| Backdoor | training data | a hidden trigger |
| Extraction | the API | a copy of the model |
| Membership inference | the API | was X in training |
Poisoning corrupts training data to shift behaviour. It does not need much: a small fraction of a fine-tuning set is enough to move a narrow behaviour, because fine-tuning sets are small and each example carries weight. The realistic vector is not a state actor — it is a scraped corpus, a public dataset nobody audited, or a feedback loop where user-submitted content becomes training data.
Backdoors are the targeted version. The model behaves normally except when an input contains a trigger, at which point it does the attacker’s thing. Evaluation cannot find it, because the model scores well on everything except inputs nobody thought to test. This is the attack that makes “we evaluated it and it was fine” an insufficient answer.
Model extraction rebuilds a model by querying it. Systematic queries plus the outputs give you a training set, and distilling on it produces a cheaper model that approximates the original. Returning log-probabilities or full confidence vectors makes this dramatically more efficient.
Membership inference asks whether a specific record was in the training data. It matters under GDPR and in any regulated setting: it turns a model into a disclosure channel about its training set, which may be patient records.
Where poisoning actually enters
Be concrete in an interview; the abstract version sounds paranoid and the concrete one sounds experienced.
- RLHF and feedback loops. Thumbs-up data that trains the next version is attacker-writable by construction. Anyone with an account can vote.
- Scraped or public corpora. A dataset assembled from the web includes whatever was on the web, including content authored to be trained on.
- Third-party fine-tuning sets and model checkpoints downloaded from a hub. A checkpoint is code plus weights, and loading a pickle-serialised checkpoint is arbitrary code execution — see Pickle, YAML, and Deserialization Attacks (Python-Specific).
- RAG corpora. Not training, but the same shape: content that enters the index shapes answers. A wiki anyone can edit is a poisoning surface with no retraining required, and it is the version most teams actually face.
Defences that are engineering, not research
# Provenance on every training record. Without this you cannot do
# the one thing that matters after an incident: find and remove.
@dataclass(frozen=True)
class TrainingRecord:
content_hash: str # dedupe and integrity
source: str # "user_feedback" | "vendor_x" | "curated"
submitted_by: str | None # attributable
ingested_at: datetime
reviewed_by: str | None # None = unreviewed- Provenance and immutable datasets. Version the dataset, hash it, and record where every record came from. When something is found, you need to answer “what else came from that source” in minutes.
- Do not train on unreviewed user content. Sample and review, or restrict training data to curated sources and use feedback only as a signal.
- Verify checkpoints. Pin by hash, prefer safetensors over pickle, and load untrusted weights in a sandbox.
- Anomaly-scan the data, not just the model — near-duplicate clusters, outliers in embedding space, records from one submitter in volume.
- Hold out a trusted eval set built entirely from curated data, so a poisoned training set cannot also poison the measurement.
Defences against extraction
These are ordinary API design, which is the point worth making.
- Return the minimum. Labels, not full probability vectors; no log-probabilities on a public endpoint. This is the single largest lever.
# The same prediction, two disclosure levels. The second is a free
# training set for whoever is collecting it.
def public(pred):
return {"label": pred.argmax_label} # minimal
def internal(pred):
return {"label": pred.argmax_label,
"scores": pred.all_class_scores, # leaks the
"logprobs": pred.logprobs} # decision surface- Rate-limit per principal, and watch for the access pattern rather than the volume: systematic coverage of the input space is what extraction looks like, and it is distinguishable from normal use.
- Authenticate everything, so there is a principal to attribute to.
- Monitor for querying that looks like enumeration — high-entropy inputs, uniform coverage, low repeat rate.
- Watermarking exists for proving a copy was derived from your model, but it is a detection and legal tool, not prevention.
For membership inference specifically the mitigation is differential privacy in training, which costs accuracy — a real trade, and the honest answer is that most teams accept the risk rather than pay it, and instead avoid training on data whose membership would be sensitive.
Where it breaks in production
- Feedback becomes training data by default. Someone wires the thumbs table to the fine-tuning job and now the loop is attacker-writable.
- No dataset versioning. After an incident you cannot say what was in the model, so the only remediation is retrain from scratch.
- Confidence scores exposed “for the UI”. A percentage on a page is also an extraction accelerant.
- A model hub treated as a package registry. It has neither the signing nor the audit trail of one.
- Eval built from the same pool as training. Poison both and everything looks fine.
Related
Interview angle 4
- “What is data poisoning, and where would it realistically come from?” - corrupting training data to shift behaviour. Realistically not an attacker breaking in, but a feedback loop where user content becomes training data, a scraped corpus nobody audited, or a third-party fine-tuning set. Follow-up: how much data do they need? For a small fine-tuning set, very little.
- “Why can’t evaluation catch a backdoor?” - because the model behaves correctly on everything except inputs containing the trigger, and you do not know the trigger. Evaluation measures the distribution you thought of.
- “How do you make model extraction expensive?” - return labels rather than probability vectors, never expose log-probs publicly, authenticate so there is a principal, and alert on access patterns that look like systematic coverage rather than on raw volume.
- “You discover a poisoned record. What now?” - the answer is provenance. If every training record carries source, submitter and hash, you scope the blast radius and remove it. Without that, the only remediation is a full retrain from a clean set you may not have.