The computer vision task taxonomy

Most CV interview confusion is a task-definition problem. “Detect the walls” could mean four different models with four different label formats, four different metrics and four different annotation budgets. Naming the task precisely is the first thing a senior candidate does.

The five tasks

classification  detection      semantic seg   instance seg
  "kitchen"     ┌───┐ door    ░░░wall░░░      ░░wall-1░░
                │   │         ▓▓▓door▓▓▓      ▓▓door-2▓▓
                └───┘

Semantic vs instance is the distinction that gets probed. Semantic asks “which class is this pixel?” — two touching walls are one blob. Instance asks “which object is this pixel?” — two touching walls are two masks. If you need to count things, or measure them individually, you need instance.

Panoptic is the union: every pixel gets a class, and countable classes (“things”) also get an instance id, while uncountable ones (“stuff” — floor, sky, background) get class only.

Choosing the task from the requirement

Work backwards from what the downstream system consumes.

A takeoff or estimation workflow needs quantities — linear metres of wall, square metres of floor — so it needs segmentation, and then a vectorisation step to turn the mask into geometry. See Contours, polygons and vector geometry.

Detection is cheaper than segmentation on every axis: annotation, training, inference and evaluation. Reach for it whenever a box is enough, and be prepared to say why a box is not enough when it isn’t — a diagonal wall’s bounding box is mostly empty, so a box tells you nothing about its length.

Images as arrays, and the conventions that bite

import cv2
import numpy as np

img = cv2.imread("plan.png")     # HWC, uint8, BGR - not RGB
h, w = img.shape[:2]             # rows first, then columns
patch = img[y0:y1, x0:x1]        # numpy indexes (row, col) = (y, x)

Three conventions cause most of the bugs in a CV codebase, and all three are worth stating out loud:

• OpenCV reads BGR, most other libraries assume RGB. A model trained on RGB and fed BGR does not crash; it just degrades, which is much worse.
• NumPy is (row, col) = (y, x), while almost every geometry API is (x, y). Every conversion between the two is a place to get it backwards.
• Torch is CHW, OpenCV is HWC. The transpose is easy; forgetting it gives a confident shape error only sometimes.

Normalisation is the fourth: a model expects a specific mean and standard deviation, and feeding raw uint8 in [0, 255] where [0, 1] normalised was expected silently destroys accuracy.

Where it breaks in production

• Class imbalance is the default. In a floor plan, “wall” pixels might be 3% of the image. Accuracy of 97% is what you get by predicting background everywhere, which is why pixel accuracy is never the metric — see Evaluating vision models and pipelines.
• Annotation disagreement caps your score. If two annotators agree at IoU 0.85, a model scoring 0.85 has reached the ceiling of the labels, and further training measures noise.
• Resolution decides what is detectable. An object smaller than the model’s stride at the input resolution cannot be found. Downscaling a 4000px drawing to 640px erases the door symbols you were looking for.
• The domain gap. A model trained on photographs transfers poorly to line drawings — different statistics, no texture, no lighting. Pretrained backbones still help, but expect to fine-tune.

Related

• OpenCV and classical preprocessing - the classical operations that come before any model.
• Object detection - detection architectures.
• Segmentation architectures - segmentation architectures.
• Evaluating vision models and pipelines - IoU, mAP, Dice, and why pixel accuracy lies.
• Convolutional networks - convolution, receptive fields, and the backbone every model here uses.