AI & ML / Computer vision / 08_ocr_in_cv_pipelines.md

OCR inside a vision pipeline

Updated 4 interview angles 5 min read source
On this page7
  1. Detect, then recognise
  2. Text versus symbol
  3. Preprocessing carries the accuracy
  4. Parsing what you read
  5. Where it breaks in production
  6. Related
  7. Interview angle

OCR inside a vision pipeline

OCR on a document and OCR on a drawing are different problems. A document is a page of text in reading order; a drawing is a mostly-graphical image with text scattered across it at arbitrary angles, in short fragments, mixed with symbols that are not text at all.

The engine comparison — Tesseract, PaddleOCR, docTR, cloud services — is in Structured Document Pipelines — Parsing, Tables, OCR, Extraction. This note is what changes when OCR is a stage in a CV pipeline.

Detect, then recognise

The two-stage structure is the thing to understand, because it is what makes neural OCR work on drawings at all.

text 
image -> text detector -> oriented boxes -> recogniser -> strings
                            (per fragment)

A detector finds where text is, usually as rotated boxes or quadrilaterals. A recogniser reads each crop independently. Because each fragment is cropped and rotated upright before recognition, vertical dimension labels and text along a diagonal wall work — which is exactly where a classical engine expecting horizontal lines fails.

Running a whole drawing through Tesseract in one call performs badly for a structural reason, not a tuning one: it assumes page layout, tries to find columns and reading order in something that has neither, and produces interleaved garbage. Crop first.

python 
# Detect regions with your own detector, recognise each crop.
for box in text_boxes:                       # rotated rect per fragment
    crop = crop_rotated(img, box)            # warp upright
    text, conf = recognise(crop)
    if conf >= 0.6:
        yield box, text, conf

Text versus symbol

A drawing contains glyphs that are not text: door swings, north arrows, hatching, electrical symbols. OCR will read some of them as characters, confidently — a door swing becomes a C, hatching becomes IIII.

Filtering is the job, and the useful filters are cheap:

  • Confidence. Necessary, not sufficient; symbol misreads are often confident.
  • Character-set plausibility. A dimension is digits, a decimal separator and maybe a unit. A room label is letters. Reject strings outside the expected alphabet for their region.
  • Geometry. Text fragments in a given drawing cluster around a small set of heights. A “word” three times the modal glyph height is a symbol.
  • Detector class. If a symbol detector already ran, mask its boxes out before OCR rather than trying to filter afterwards.

Preprocessing carries the accuracy

More than the engine choice does. In rough order of effect:

Step Why it matters
Deskew recognisers assume upright input
Upscale small text needs pixels
Binarise adaptively uneven scans, faint linework
Remove long lines rulings merge into characters

Upscaling counterintuitively helps. Recognisers have a preferred input height, typically in the tens of pixels; text below it is unreadable and resampling up by 2-4x with a smooth interpolation genuinely improves accuracy even though no information was added.

Removing long lines before OCR is the drawing-specific trick. A wall line passing through a label merges with the glyphs and corrupts them. Detect long runs morphologically and subtract them:

python 
import cv2

horiz = cv2.getStructuringElement(cv2.MORPH_RECT, (40, 1))
lines = cv2.morphologyEx(binary, cv2.MORPH_OPEN, horiz)
text_only = cv2.subtract(binary, lines)

The kernel length must exceed the widest glyph but stay below the shortest wall run, and it scales with resolution — see OpenCV and classical preprocessing.

Parsing what you read

Recognised strings are not values. 3 600 , 3,600, 3.60m and 3600 may all mean the same length, and a decimal separator misread as a comma changes a measurement by a factor of a thousand.

python 
import re

DIM = re.compile(r"^(\d{1,3}(?:[ ,.]\d{3})*|\d+)(?:[.,](\d+))?\s*(mm|cm|m)?$")

def parse_dimension(s, default_unit="mm"):
    m = DIM.match(s.strip())
    if not m:
        return None
    whole = re.sub(r"[ ,.](?=\d{3}\b)", "", m.group(1))
    value = float(f"{whole}.{m.group(2)}" if m.group(2) else whole)
    return value, m.group(3) or default_unit

Then sanity-check against geometry. If OCR says a wall is 3600mm and the rectified pixel length times the scale factor says 3580mm, that agrees. If it says 36000mm, the reading is wrong — or the scale is. Cross-checking two independent estimates is the single most valuable thing in this stage, and it is what turns OCR from a guess into a measurement.

Where it breaks in production

  • Rotated text read as gibberish. The detector found it; the recogniser got it un-rotated. Warp each crop upright using the box angle.
  • Confidence is per-engine and not comparable. A 0.7 from one engine is not a 0.7 from another. Calibrate the threshold per engine on your own data.
  • Language and character-set defaults. An engine set to English on a drawing full of digits will hallucinate letters into numbers: 0 to O, 1 to l, 5 to S. Restrict the charset per region type.
  • Cloud OCR sends your data out. For client drawings under NDA this is a contractual question before it is a technical one.
  • Per-call latency at scale. One call per fragment on a drawing with 800 labels is 800 calls. Batch, and cache by crop hash.

Interview angle 4

  • “Why not just run Tesseract on the whole drawing?” - it assumes page layout and reading order, which a drawing does not have, and it needs upright text, which drawings do not have either. Detect text regions first, warp each crop upright, recognise per fragment.
  • “OCR reads door swings as letters. Fix it.” - confidence alone will not do it, because symbol misreads are often confident. Filter on character-set plausibility for the region type, on glyph height against the drawing’s modal height, and ideally mask out boxes a symbol detector already claimed.
  • “How do you know a dimension you read is correct?” - cross-check it against geometry. Rectified pixel length times scale should agree with the OCR’d value within tolerance. Two independent estimates agreeing is worth far more than one confident one.
  • “Small text is unreadable.” - upscale 2-4x before recognition. Recognisers have a preferred input height and adding pixels helps even though it adds no information.
Previous
Contours, polygons and vector geometry