Monocular depth estimation

Recovering depth from a single image is geometrically impossible — a small close object and a large far one project identically. Monocular depth models work anyway because they learn priors: perspective, familiar object sizes, texture gradients, occlusion ordering. Knowing that it is a learned prior rather than a measurement is the whole interview.

Relative, affine-invariant, metric

The single most important distinction, and the one candidates get wrong.

MiDaS produces affine-invariant inverse depth. Its output is proportional to 1/distance up to an unknown scale and shift, which means you can say pixel A is nearer than pixel B, but the raw values are not metres and the ratio between two of them is not a ratio of distances.

Depth Anything follows the same family and pushed quality substantially by training on a very large corpus of pseudo-labelled images. It ships in relative and metric-finetuned variants — check which one you loaded, because they are not interchangeable and both return a plausible-looking array.

To get metres you need an anchor. Two known depths let you solve the scale and shift; a known object size or a calibrated sensor does the same. Without one, any number you report in metres is invented.

import numpy as np

def to_metric(inv_depth, anchors):
    """anchors: [(y, x, metres), ...] - at least two, spread apart.

    MiDaS-style output is affine-invariant inverse depth, so fit
    d_metric = 1 / (a * inv + b) by least squares on 1/metres.
    """
    obs = np.array([inv_depth[y, x] for y, x, _ in anchors])
    tgt = np.array([1.0 / m for *_, m in anchors])
    a, b = np.polyfit(obs, tgt, 1)
    return 1.0 / np.clip(a * inv_depth + b, 1e-6, None)

When depth helps, and when it does not

For a floor plan or an architectural drawing, monocular depth is usually the wrong tool — the subject is a flat sheet of paper, so the true depth map is a plane, and a homography already solves the geometry exactly. Reaching for a depth model there signals you have not thought about the problem.

Where it earns its place:

• Photographs of real rooms, where you want approximate layout, floor extent, or a rough sense of scale from a phone image.
• Occlusion ordering — which object is in front — for compositing or for filtering detections that cannot be where they appear to be.
• Ground-plane estimation, then projecting detections onto it for a rough top-down view.
• Segmentation assistance, where a depth discontinuity marks a boundary that has no colour contrast.

The honest framing for an interview: depth from a single image is a strong prior and a weak measurement. Use it to rank and filter, not to quantify, unless you have anchored it.

The pipeline, and its cost

import torch

model = torch.hub.load("intel-isl/MiDaS", "DPT_Hybrid").eval()
# Preprocessing must match the checkpoint - the transform ships with it.
with torch.inference_mode():
    inv_depth = model(batch)          # (N, H', W'), not metres

Three practical notes that come up:

• Output resolution is lower than input. You upsample back, and the boundaries are soft. Depth edges do not align pixel-perfectly with object edges, so never use a depth discontinuity as a precise boundary.
• It is not temporally stable. Run it per frame on video and the depth flickers, because nothing constrains consecutive frames. Smooth temporally, or use a model trained for video.
• Cost is real. A transformer depth model is heavier than the detector in front of it. Run it once per scene, not per detection.

Where it breaks in production

• Flat, textureless surfaces. A blank wall gives the model nothing to reason from, and it falls back on layout priors that may be wrong.
• Reflections and glass. Predicted depth follows the reflected scene, so a mirror reads as a doorway into another room.
• Unusual viewpoints. Trained mostly on eye-level photographs; a ceiling or floor-level shot leaves the training distribution.
• Scale drift across images. Each image gets its own unknown scale, so two photographs of the same room are not comparable without re-anchoring. This breaks anyone who averages depth across a set.
• Confident output on impossible input. Feed it a line drawing and you get a smooth, plausible, meaningless depth map. There is no “I don’t know”.

Related

• Geometry: homography, rectification, vanishing lines - the exact answer when the scene is planar.
• Floor plan recognition and takeoff - why depth is usually not the tool there.
• Segmentation architectures - depth as an extra input channel.
• CV pipelines and designing for imperfect input - gating on whether an estimate should be trusted at all.