AI & ML / Classical ML / 10_anomaly_detection.md

Anomaly detection

Updated 6 interview angles 5 min read source
On this page8
  1. First: is it actually unsupervised?
  2. The methods, and what each assumes
  3. Point, contextual, collective
  4. Evaluating without labels
  5. In production
  6. Rules still earn their place
  7. Related
  8. Interview angle

Anomaly detection

The problem where you mostly do not have labels, the positive class is 0.1% of the data, and the definition of “anomalous” changes under you. Fraud, intrusion, equipment failure and data quality all land here.

First: is it actually unsupervised?

The most valuable question, and the one that separates a considered answer.

You have Do this
Confirmed labels supervised classification
A few labels semi-supervised, or PU learning
No labels unsupervised detection

Teams reach for Isolation Forest when they have 400 confirmed fraud cases, which is enough to train a classifier that will beat any unsupervised method. Unsupervised is the fallback, not the default — the honest framing is “what labels can I get?” before “which detector?”.

The related trap: fraud labels arrive late. A chargeback lands 60 days after the transaction, so your training set’s recent rows are labelled “not fraud” only because nobody has noticed yet. That is label leakage in reverse and it inflates every offline number.

The methods, and what each assumes

Method Assumes Good at
Isolation Forest anomalies are few and separable tabular, general default
Local Outlier Factor density varies locally clusters of differing density
One-Class SVM a boundary exists small, clean data
Autoencoder normal is reconstructable images, high dimensions
Robust z-score / IQR roughly unimodal one column, quick checks

Isolation Forest is the right first try on tabular data. It isolates points with random splits, and anomalies need fewer splits to isolate — so it scales linearly and needs no distance metric.

python 
from sklearn.ensemble import IsolationForest

iso = IsolationForest(
    contamination=0.01, random_state=0
).fit(X_train)

# Higher score = more anomalous.
scores = -iso.score_samples(X_test)

Gotcha: contamination is not a discovery, it is an assumption you are supplying. Set it from your operational capacity — how many alerts can a human review per day — not from a guess about the true rate.

Use the score, not the binary label. Every real deployment ranks and thresholds by capacity, and a hard predict() throws away the ordering you need to do that.

Point, contextual, collective

The taxonomy that stops you using the wrong method:

  • Point — a single record is odd on its own. A £50,000 transaction.
  • Contextual — odd given the context. £200 is normal, at 4am from a new country it is not. Requires the context in the features.
  • Collective — no single point is odd, the sequence is. A thousand £9 transactions in an hour.

Isolation Forest finds point anomalies well and collective ones not at all. Catching the third kind means aggregating first — features over a window per entity — which is a feature-engineering decision, not a model choice.

Evaluating without labels

The hard part, and where most projects quietly fail.

  • Precision@k. Take the top 100 scores, have a human check them. It is the number the business cares about, because it maps to reviewer time.
  • PR-AUC over ROC-AUC whenever you do have labels. At 0.1% positives, ROC looks excellent while the model is useless — see ROC-AUC vs PR-AUC.
  • Injected anomalies. Synthesise known-bad records and measure recall on them. Weak evidence, but it catches a detector that has stopped working.
  • Time to detection, for anything streaming. Catching it eventually is not the same as catching it in time.

In production

The threshold drifts. Normal behaviour changes with seasons, releases and pricing. A fixed threshold that alerted 20 times a day in January alerts 400 in July. Recalibrate on a rolling window, and alert on the alert rate as a health signal for the detector itself.

Threshold as a budget, not a constant — the reviewers can take 50 a day, so that is what the quantile is derived from:

python 
import numpy as np

BUDGET = 50  # alerts a reviewer can clear daily

recent = scores[-30 * DAILY_VOLUME:]
q = 1 - BUDGET / DAILY_VOLUME
threshold = np.quantile(recent, q)

fired = (scores_today > threshold).sum()

Track fired over time. A sudden collapse to zero usually means a broken feature pipeline, not a quiet week.

Alert fatigue is the failure mode, not model accuracy. A detector firing 200 times a day with 5% precision trains its reviewers to close alerts without reading them, which is worse than no detector. Tune to the review capacity you actually have.

An adversary adapts. Fraud is not a stationary distribution; it responds to your detector. That argues for retraining cadence, for keeping some rules alongside the model, and for not publishing your features.

Rules still earn their place

A hybrid is usually right: deterministic rules for the known-bad patterns you must catch and can justify, a model for the long tail. Rules are explainable in a dispute and instant to change when the regulator asks; the model finds what nobody wrote a rule for.

Interview angle 6

  • “How would you detect fraud?” - first ask whether it is really unsupervised. A few hundred confirmed cases beat any unsupervised detector, so the question is what labels you can get. Unsupervised is the fallback, not the default.
  • “What’s the catch with fraud labels?” - they arrive late. A chargeback lands 60 days later, so recent rows are labelled negative only because nobody has noticed yet. That inflates every offline metric unless you hold back a maturation window.
  • “Which algorithm?” - Isolation Forest first on tabular data: linear time, no distance metric, and anomalies isolate in fewer splits. Use the score rather than the binary label, because deployment always means ranking and thresholding by review capacity.
  • “What does contamination do?” - it is an assumption you supply, not something learned. Set it from how many alerts a human can review per day rather than from a guess at the true rate.
  • “How do you evaluate with no labels?” - precision@k with human review of the top scores, PR-AUC rather than ROC-AUC once you have any labels, and injected synthetic anomalies as a regression check. At 0.1% positives, ROC-AUC looks excellent while the model is useless.
  • “What actually kills these systems?” - alert fatigue and threshold drift, not model accuracy. A detector firing 200 times a day at 5% precision teaches reviewers to close alerts unread, which is worse than not having it.
Previous
Dimensionality reduction