AI & ML / Computer vision / 09_cv_pipelines_and_robustness.md

CV pipelines and designing for imperfect input

Updated 4 interview angles 5 min read source
On this page8
  1. The pipeline as stages with contracts
  2. Fallback ladders
  3. Validate with independent checks
  4. Reproducibility
  5. Handling scale: tiling
  6. Where it breaks in production
  7. Related
  8. Interview angle

CV pipelines and designing for imperfect input

A demo takes a clean image and returns a good answer. A product takes whatever the user uploads — a phone photo of a creased printout at an angle, a 300 MB scan, a PDF that is secretly a single JPEG — and must return either a good answer or an honest failure. Designing that gap is most of the job.

The pipeline as stages with contracts

text 
ingest -> normalise -> rectify -> detect/segment
       -> vectorise -> measure -> validate -> emit
              │
              └─ any stage may abstain

Give each stage an explicit output contract that includes a confidence and a reason, not just a result. A stage that can only succeed forces every failure downstream, where it appears as a wrong number rather than a caught problem.

python 
@dataclass(frozen=True)
class StageResult:
    value: Any | None
    confidence: float          # 0-1, calibrated per stage
    method: str                # which path produced it
    warnings: tuple[str, ...] = ()

    @property
    def ok(self) -> bool:
        return self.value is not None

method matters more than it looks. When you have three fallback paths, the field that says which one ran is what makes production debuggable and what lets you measure how often the good path is actually taken.

Fallback ladders

For each stage, order the strategies from most precise to most general, and descend on failure. The ladder for rectification:

Attempt Trigger to descend
Detect page border border not found
Vanishing points too few parallel lines
Assume already flat always succeeds

Two rules make a ladder work rather than paper over problems:

  • Every descent is recorded, so “we fell back” is a metric you can alert on. A jump in fallback rate is usually an upstream change, not a model change.
  • The last rung must be honest. If the final fallback is “assume it is already rectified”, the output carries a warning and a lower confidence, and downstream measurement either refuses or flags the result. Silently returning the un-rectified image as if it were rectified is worse than failing.
python 
def rectify(img):
    for name, fn in (("border", by_border),
                     ("vanishing", by_vanishing_points),
                     ("identity", lambda i: (i, np.eye(3)))):
        try:
            out, H, conf = fn(img)
        except RectifyError:
            continue
        if conf >= THRESHOLDS[name]:
            return StageResult(out, conf, method=name)
    return StageResult(None, 0.0, method="none", warnings=("no rectify",))

Validate with independent checks

The strongest robustness technique is checking a result against something computed a different way. Disagreement is a signal you can act on; a single estimate gives you nothing.

  • Areas must sum. Room areas plus wall footprint plus circulation should approximate the building footprint. A 15% shortfall means a missed region.
  • OCR against geometry. A read dimension should match pixel length times scale. See OCR inside a vision pipeline.
  • Topology. Rooms should not overlap; a wall should border something on both sides. Overlap above a tolerance means a segmentation error.
  • Priors. A door 4 metres wide, a ceiling 15 metres high, a 400 m² bathroom — cheap range checks catch a surprising share of real failures.

Each check is a few lines and each one catches a failure class that no model metric surfaces.

Reproducibility

An experiment you cannot re-run is a rumour. The minimum that makes a prototype reproducible:

python 
CONFIG = {                      # one dict, serialised into every output
    "pipeline_version": "2026.08.3",
    "weights_sha": "a1b2c3d4",
    "input_size": 1280,
    "conf_threshold": 0.35,
    "scale_source": "scale_bar",
}
  • Pin the weights by hash, not by filename. best.pt is not a version.
  • Seed everything, and accept that GPU nondeterminism means bitwise reproduction still is not guaranteed — the same caveat as Determinism and reproducible LLM tests.
  • Share one preprocessing function between training and serving. Divergence here is the most common cause of a model that scores well and performs badly.
  • Store the config with the output. When a customer disputes a number six months later, the only useful artefact is the exact configuration that produced it.

Handling scale: tiling

Large drawings do not fit in a model’s input, and downscaling erases the small symbols you need.

python 
def tiles(h, w, size=1024, overlap=128):
    step = size - overlap
    for y in range(0, max(h - overlap, 1), step):
        for x in range(0, max(w - overlap, 1), step):
            yield y, x, min(y + size, h), min(x + size, w)

The overlap must exceed the largest object you expect, or an object is cut in every tile that contains it. Merge detections across seams with NMS in global coordinates; for segmentation, blend logits in the overlap rather than argmaxing per tile, which leaves visible seams.

Where it breaks in production

  • Thresholds tuned once. A confidence threshold set on the first fifty files drifts as input mix changes. Monitor the distribution, not just the value.
  • No abstention path. A pipeline that always answers will answer wrongly. The ability to say “this scan is too poor” is a feature.
  • Memory on large inputs. A 12000x9000 PDF page decodes to several hundred MB; two in flight in one worker is an OOM. Cap dimensions at ingest and stream tiles.
  • PDF is not an image format. Vector PDFs should be rendered at chosen DPI, not screenshotted; some contain a raster scan, some contain real geometry you could extract directly and skip CV entirely. Check before running a model.
  • The happy path is over-tested. Keep a fixture set of genuinely bad inputs — blurred, rotated, partial, wrong document type — and assert the pipeline degrades rather than crashes.

Interview angle 4

  • “How do you handle inputs your model was not trained for?” - a fallback ladder per stage, ordered most precise to most general, where every descent is recorded as a metric and the final rung returns a warning and a reduced confidence rather than a confident guess. Abstention is a feature.
  • “How do you know a measurement is right when there is no ground truth?” - independent cross-checks. Areas that must sum, OCR against pixel geometry, topology that must hold, and range priors. Two estimates agreeing is evidence; one estimate is not.
  • “Your drawing is 12000px wide and the model takes 1024.” - tile with overlap larger than the biggest object, run per tile, merge with NMS in global coordinates, blend logits rather than masks across seams. Downscaling instead would erase the small symbols.
  • “How do you make a prototype reproducible?” - pin weights by hash, one config dict serialised into every output, one shared preprocessing function between train and serve, and seeds set while being honest that GPU nondeterminism still prevents bitwise reproduction.
Previous
OCR inside a vision pipeline