Design: a fraud detection system

The canonical classical-ML system design. It exercises imbalance, cost-sensitive thresholds, real-time features, label delay and adversarial drift — most of the hard parts in one problem.

Clarify

• Decision: block, allow, or send to manual review. Three outcomes, not two.
• Scale: 10k transactions/second peak.
• Latency: synchronous, inside the payment flow — under 100ms.
• Base rate: ~0.1% fraudulent.
• Costs: a missed fraud averages $500; a false positive costs a review (~$5) and customer friction.
• Labels: chargebacks arrive 30-90 days later. Manual reviews give faster partial labels.

The cost asymmetry (100:1) and the label delay are the two facts that drive everything downstream.

Frame it

Binary classification producing a score, not a decision. The score maps to three actions by two thresholds:

score < 0.3          -> allow
0.3 <= score < 0.85  -> manual review queue
score >= 0.85        -> block

Thresholds come from the cost matrix and the review team’s capacity, not from F1. If the team can review 2,000 cases/day, that caps the middle band regardless of what the model says. See Precision, recall and F1.

Metric: PR-AUC, not ROC-AUC. At 0.1% positives the true-negative pool is enormous, so a large increase in false positives barely moves FPR while precision collapses. See ROC-AUC vs PR-AUC.

Architecture

transaction -> feature service ──> model ──> score ──> decision
                    │                                     │
        ┌───────────┴────────────┐                        v
   online store            streaming aggregates      allow / review / block
   (Redis: profile,        (Flink: velocity                    │
    card, merchant)         counters, last 5m)                 v
                                                        review outcomes
                                                        (fast labels)

Features

Grouped by how they’re computed, because that determines the infrastructure:

Velocity features are the most predictive and the hardest to serve. “Five transactions in the last two minutes” needs a streaming pipeline (Kafka plus Flink) writing counters to the online store — you cannot compute it from a warehouse at request time.

Point-in-time correctness is essential. Training rows must use feature values as of that transaction’s timestamp. Joining current profile state onto historical transactions leaks the outcome backwards. This is where a feature store earns its cost. See Feature stores and point-in-time correctness.

Model

Gradient boosting — LightGBM or XGBoost. It wins on tabular data, trains fast enough for frequent retraining, handles missing values natively, and inference is well under the latency budget. See Gradient boosting.

Handle imbalance with scale_pos_weight plus threshold tuning, not SMOTE. Note that re-weighting distorts probabilities, so recalibrate if you use the score in an expected-loss calculation. See Imbalanced data.

Keep a rules layer alongside the model. Rules encode known patterns instantly, are explainable to compliance, and cover the model’s blind spots on brand-new attack patterns:

if velocity_5m > 10 or amount > user.p99_amount * 10:
    # rules fire first, deterministic
    return BLOCK
return model_decision(score)

Rules-plus-model is the production reality, and proposing only a model is a gap.

Latency

100ms budget, roughly:

feature fetch (online store, parallel)   ~20ms
derived features                          ~5ms
model inference                          ~10ms
rules + decision                          ~5ms
logging (async, off the path)              0ms
                                     ---------
                                        ~40ms   comfortable

Fetch features in parallel, not sequentially. Have a fallback: if the feature store times out, score on transaction-only features and lean on rules. A payment flow cannot hang because the model is unavailable — degraded is better than down.

The label delay problem

Chargebacks take 30-90 days. Consequences:

• You cannot evaluate today’s model on today’s data. Accept a 30-90 day lag on the true metric.
• Manual review outcomes are fast partial labels — biased toward the reviewed band, but timely. Weight accordingly.
• Monitor proxies meanwhile: score distribution, feature drift, block rate, review-queue depth. See Monitoring and drift.

Adversarial drift

Unlike most ML problems, the data-generating process is an adversary who adapts to your model.

• Retrain frequently — weekly or faster.
• Expect concept drift by default, not as an anomaly.
• Don’t expose scores or reasons to users; that’s a feedback channel for attackers.
• Keep a rules layer for rapid response — a new attack pattern can be blocked in an hour, retraining takes days.
• Watch for blocked-transaction blind spots: you never learn whether blocked transactions were actually fraud. Consider letting a tiny random sample through to maintain an unbiased signal, if the business tolerates it.

That last point is a genuine feedback-loop problem and a strong thing to raise.