AI & ML / Computer vision / 10_cv_evaluation.md

Evaluating vision models and pipelines

Updated 5 interview angles 5 min read source
On this page8
  1. IoU is the base unit
  2. mAP, and what the “mean” hides
  3. Choosing the metric from the failure that costs you
  4. Splitting the data honestly
  5. Evaluate the pipeline, not just the model
  6. Where it breaks in production
  7. Related
  8. Interview angle

Evaluating vision models and pipelines

Pixel accuracy is the metric that makes a useless model look excellent. Every metric here exists because the naive one lies in a specific way, and being able to say which lie each one fixes is the interview.

IoU is the base unit

python 
def iou(a, b):
    """Boxes as (x1, y1, x2, y2), or boolean masks."""
    inter = intersection_area(a, b)
    return inter / (area(a) + area(b) - inter)

Intersection over union: the overlap divided by the combined extent. It ranges 0 to 1, it is symmetric, and it penalises both missing the object and overshooting it — which is why it beats “did the centre land inside”.

Dice is the other overlap metric, 2|A∩B| / (|A| + |B|). Dice and IoU rank predictions identically — they are monotonically related — so a model better on one is better on the other. Dice reads higher for the same prediction, which is why comparing a Dice number to an IoU number across two papers is meaningless.

Segmentation convention leans Dice, detection leans IoU. Say which you are quoting, always.

mAP, and what the “mean” hides

For detection, a prediction is a true positive if it matches a ground-truth box of the same class above an IoU threshold. Sweep the confidence threshold, get a precision-recall curve, take the area under it — that is average precision for one class. Mean it over classes and you have mAP.

Notation Means
mAP@0.5 one threshold, lenient
mAP@0.5:0.95 averaged over ten thresholds
mAP small only small objects

mAP@0.5:0.95 is much stricter because it demands tight localisation, not just detection. A model can score 0.85 at 0.5 and 0.45 averaged, and the gap tells you the boxes are loose. Quoting mAP@0.5 alone when the number is flattering is a thing interviewers watch for.

The mean over classes hides imbalance: nine classes at 0.9 and one at 0.1 gives 0.82, and the one at 0.1 might be the class the product depends on. Always report per-class AP alongside the mean.

Choosing the metric from the failure that costs you

If the cost is Watch
Missing a thing recall, false negatives
False alarms precision
Wrong measurement area error, not IoU
Bad boundaries boundary F1

This is the part that separates candidates. For a takeoff workflow the business metric is not IoU at all — it is percentage error on the quantity. A mask can score IoU 0.92 and still produce a room area 8% too large if the error is systematically on the outside of the boundary. IoU is symmetric; area error is signed, and the sign is what tells you the model is consistently over-segmenting.

python 
def area_error(pred_mask, true_mask, px_to_m2):
    p, t = pred_mask.sum() * px_to_m2, true_mask.sum() * px_to_m2
    return (p - t) / t          # signed - direction is the diagnosis

Boundary F1 measures agreement within a tolerance band of the true boundary, ignoring the interior. Use it when boundary precision is the product and the interior is trivially correct, which is the usual case for large regions.

Splitting the data honestly

The most common way a CV evaluation lies is leakage.

  • Split by document or site, not by image. Five pages of one drawing set in train and one in val means the model has seen that draughtsman’s conventions, that scanner, that line weight.
  • Beware near-duplicates. Re-scans, revisions and crops of the same source are effectively the same image. Hash-based deduplication before splitting.
  • Stratify by the rare class, or a random split gives a val set containing two examples of the class you care about, and its AP becomes noise.
  • Keep a held-out set you never tune on. Val gets used for threshold selection and early stopping, which is a form of fitting.

Evaluate the pipeline, not just the model

The model metric and the product metric are different numbers, and only the second one matters.

text 
model IoU 0.93
  -> vectorised, simplified   (loses 1%)
  -> scale from OCR'd bar      (wrong 3% of the time)
  -> area in m2                (final error: 6% median, 40% p95)

Instrument end to end. A per-stage error budget tells you where to spend effort; without it, teams optimise the model because that is the part with a leaderboard, while the scale-detection step contributes most of the error.

Track the fallback rate from CV pipelines and designing for imperfect input as a first-class metric. A pipeline whose accuracy is stable while its fallback rate climbs is degrading, and no model metric shows it.

Where it breaks in production

  • Pixel accuracy quoted on imbalanced masks. 97% by predicting background.
  • Val used for tuning, then reported. The number is optimistic by however many thresholds you tried.
  • Annotation ceiling ignored. If annotators agree at IoU 0.85, a model at 0.85 has converged to the label noise, and further gains are overfitting.
  • A single aggregate number. Report the distribution. A median error of 3% with a p95 of 60% is a very different product from a uniform 8%.
  • No slice analysis. Aggregate metrics hide that scanned inputs perform fine and photographed ones do not. Slice by source, resolution, and quality.

Interview angle 5

  • “Your segmentation is 97% accurate.” - pixel accuracy on a 3%-foreground mask is what you get from predicting background everywhere. Ask for IoU or Dice per class, and the per-class breakdown rather than the mean.
  • “IoU or Dice?” - they rank identically, being monotonically related, so it does not change which model wins. Dice reads higher for the same prediction, so never compare a Dice number to an IoU one. Segmentation convention leans Dice, detection leans IoU.
  • “mAP@0.5 is 0.88, mAP@0.5:0.95 is 0.44. What does that tell you?” - the model finds the objects but localises them loosely. If downstream measurement depends on box extent, the second number is the one that matters.
  • “IoU is 0.92 but the customer says areas are too big.” - IoU is symmetric, so it cannot see that the error is systematically outside the boundary. Track signed area error; the sign diagnoses over-segmentation directly.
  • “How do you split a dataset of scanned drawings?” - by document or site, never by image, and deduplicate near-identical re-scans first. Otherwise the model has seen the same draughtsman and scanner in training.
Previous
CV pipelines and designing for imperfect input