Object detection

Detection predicts a set of boxes with class labels and confidences. The architecture question (“YOLO or Detectron2?”) is much less interesting than the mechanics underneath, which is where interviews go: how a fixed-size network output becomes a variable-length set of boxes, and how duplicates get removed.

One-stage vs two-stage

Two-stage first proposes regions likely to contain anything, then crops features for each and classifies them. The crop step is what buys accuracy on small and overlapping objects, and it is what costs the latency.

One-stage predicts class and box directly at every spatial location in one pass. YOLO is the family that made this practical, and for most production work it is the default: fast enough for video, accurate enough, with a mature training and export toolchain.

DETR and its successors are a third shape: a transformer that predicts a fixed set of queries and matches them to ground truth with bipartite matching, which removes the need for NMS entirely. Worth naming — it changes the answer to “how do you handle duplicates”.

Anchors, and why they mostly went away

Early detectors placed a grid of prefabricated boxes (“anchors”) of assorted scales and aspect ratios, and learned an offset from the nearest anchor to each true box. This meant tuning anchor sizes to your dataset — an extra hyperparameter that silently caps recall if your objects are shaped unlike COCO’s.

Anchor-free detectors predict the box directly from the centre point, which removes that tuning step. Most current one-stage detectors are anchor-free.

The useful interview point: if you are using an anchor-based detector on unusual geometry — very long thin walls, tiny symbols — check the anchor configuration before blaming the model. A default anchor set assumes roughly square, photograph-scale objects.

NMS is where duplicate boxes die

A dense detector fires at several nearby locations for the same object. Non-maximum suppression keeps the highest-confidence box and drops the rest that overlap it beyond a threshold.

import numpy as np

def nms(boxes, scores, iou_thresh=0.5):
    """boxes: (N, 4) as x1, y1, x2, y2. Returns kept indices."""
    order = scores.argsort()[::-1]
    keep = []
    while order.size:
        i = order[0]
        keep.append(i)
        if order.size == 1:
            break
        ious = box_iou(boxes[i], boxes[order[1:]])
        order = order[1:][ious <= iou_thresh]
    return keep

Two knobs, two failure modes, and both come up:

• Confidence threshold too high drops real objects; too low floods you with false positives. It trades precision against recall and belongs downstream of the model, tuned per deployment, not baked in.
• IoU threshold too low deletes genuinely overlapping objects — two doors side by side become one. Too high leaves duplicates. When objects legitimately overlap, consider Soft-NMS, which decays scores instead of deleting, or class-wise NMS so a door never suppresses a window.

Training in practice

You will almost never train from scratch. Start from COCO-pretrained weights and fine-tune; the backbone has already learned edges, corners and texture, and those transfer even to line drawings where the source domain looks nothing like the target.

from ultralytics import YOLO

model = YOLO("yolo11n.pt")           # pretrained, nano
model.train(
    data="plans.yaml",               # paths plus class names
    imgsz=1280,                      # raise for small objects
    epochs=100,
    patience=20,                     # early stop
)

Image size is the lever that matters for small objects. A symbol occupying 12 pixels at 640 occupies 24 at 1280, and detectors have a floor below which an object is smaller than the stride and simply cannot be predicted. Raising imgsz costs quadratic memory and time, which is why tiling is often better — see CV pipelines and designing for imperfect input.

Watch the label format. YOLO wants normalised centre-x, centre-y, width, height; COCO wants absolute x, y, width, height; Pascal VOC wants x1, y1, x2, y2. Every conversion is a chance to ship a model trained on scrambled boxes, and the symptom is a loss that falls and a mAP that stays near zero.

Where it breaks in production

• Class imbalance across classes. 5,000 doors and 40 fire exits means the rare class is never learned. Oversample, or weight the loss.
• Duplicate near-identical images in train and val. Scanned pages of the same building leak, val mAP looks excellent, production does not. Split by document or by site, never randomly by image.
• Confidence is not calibrated. A 0.9 score is not a 90% probability. If a downstream decision depends on the number, calibrate it.
• Objects at tile boundaries. Tiling cuts an object in half and you get two partial boxes. Overlap tiles and NMS across the seams.
• Aspect-ratio distortion. Resizing without letterboxing squashes boxes, and any measurement taken from them is wrong.

Related

• The computer vision task taxonomy - when a box is not enough.
• Segmentation architectures - when you need pixels.
• Evaluating vision models and pipelines - IoU, mAP and the precision-recall curve behind it.
• CV pipelines and designing for imperfect input - tiling, seams and fallbacks.
• Transfer learning
  • why COCO pretraining transfers to unlike domains.