Company A — Middle AI/Python Engineer — prep map

Checklist format: tick what you can explain and defend under questioning, not what you have used. Bold is the primary note; the rest add depth.

Priority: 🔴 core (in the requirements, will be probed) · 🟡 nice to have (in the ad, lighter) · 🟢 breadth (implied)

The ad, decoded

Requirement	What they will actually probe	Weight
4+ years production Python: FastAPI/async, PostgreSQL, queues, Docker	a pipeline with GPU workers, idempotent retries, a DB schema	🔴
2+ years LLMs/VLMs in production	incidents, evals, cost — things only production teaches	🔴
Open-weight VLMs: Qwen2.5-VL / Qwen3-VL, InternVL, Llama Vision, Pixtral	what each is built for, licences, failure modes with mitigations	🔴
Structured extraction: JSON-schema, constrained decoding, scans/PDFs/forms, validation	the extraction pipeline end to end, what the grammar does not guarantee	🔴
Fine-tuning: LoRA / QLoRA, PEFT	what to adapt on a VLM, data, resolution skew, merge-then-quantise	🔴
Serving: vLLM / SGLang / Triton, AWQ / GPTQ / FP8, batching, KV cache, GPU sizing vs SLA and budget	the flags, the memory arithmetic, a GPU count with assumptions	🔴
Classical CV: OpenCV, DocLayout / Surya, PaddleOCR / Tesseract, YOLO	when OCR beats the VLM; the hybrid	🟡
LangGraph, RAG, vector DBs	one solid design each	🟡
MLOps: MLflow / W&B, CI/CD for models	registry, eval gate, shadow/canary	🟡
Presales or estimation	a scoped pilot with numbers	🟡

The product this ad describes: document extraction at volume on self-hosted VLMs — scans, PDFs and forms in; validated JSON out; with a GPU bill and an SLA. Everything below is that system; the section “The design question” walks it end to end.

🔴 Production Python

FastAPI + async, done properly — DI, async def vs threadpool, background work, testingFastAPI overview Async and performance Dependencies and injection Background tasks vs Celery Testing
asyncio in anger — TaskGroup, cancellation, queues and backpressure, blocking the loopTaskGroup and structured concurrency Cancellation, timeouts, errors asyncio.Queue vs threading.Queue run_in_executor / to_thread
PostgreSQL for a document store — schema for documents/pages/extractions, JSONB, indexes, migrationsIndexes Relations and keys Async SQLAlchemy Alembic Partition lifecycle pgvector in production
Queues — Celery / RabbitMQ / SQS: retries, idempotency, DLQ, fan-out/fan-in for pagesBroker comparison Celery idempotency Celery retries Dead-letter queue Canvas pitfalls RabbitMQ exchanges, queues, DLX SQS visibility timeout
Docker for GPU services — multi-stage images, pinned weights, compose, health and readinessDocker Multi-stage builds Compose Probes and HPA
Observability for the pipeline — traces per document, metrics that page, incident responseLLM observability AI incident response Incident response and runbooks

🔴 LLMs / VLMs in production — the things only production teaches

Evals, not opinions — golden sets, regression on every change, LLM-as-judge limitsBuilding eval sets Why LLM eval is hard LLM-as-judge
Cost — tokens per page, cache, routing, batch; cost per outcomeCut LLM spend without losing quality Cost attribution Inference basics
Determinism, drift, model swaps — temperature 0 is not deterministic; pin revisionsDeterminism and reproducibility Monitoring and drift
Hosted APIs when they are the right call — structured outputs, tool loop, cachingThe Claude Messages API in practice Model selection and routing Provider abstraction and resilience

🔴 Open-weight VLMs — strengths and failure modes

How a VLM sees — encoder–projector vs cross-attention, resolution strategies, the token arithmetic (one token per 28×28 block on Qwen; an A4 scan is 1,260 tokens at the recommended budget, 11,000+ at the default)How a VLM sees an image Multimodal models KV cache
The landscape — Qwen2.5-VL / Qwen3-VL, InternVL 3/3.5, Llama 3.2 Vision (and its EU licence clause), Pixtral / Mistral Small, Gemma 3; the document-OCR VLMs; benchmarks and why DocVQA is saturatedOpen-weight VLMs: the landscape
Failure modes, each with a mitigation — hallucinated absent fields, small print lost to the pixel budget, counting and coordinates, repetition loops on tables, rotation, multi-page context, injection through image text, drift from engine and checkpoint changesVLM failure modes in production Hallucinations and tools Prompt injection

The line the ad is testing for: first-hand experience of where these models do well and where they break, not just book knowledge. Have three war stories in the shape symptom → cause → fix → what we measured after: the invented total (nullable fields + evidence quotes + arithmetic checks), the repeated table row (loop detection while streaming + split the table), the accuracy drop nobody could explain (resolution skew between fine-tune and serving).

🔴 Structured extraction

The pipeline — preprocess → classify → extract → validate → route → store; idempotent GPU workers; the review loop as the data engineStructured extraction from documents Document parsing pipelines
The schema is the product — nullable values, evidence quotes, page numbers, versioned schema stored with each rowStructured extraction from documents(§ The schema is the product)Pydantic in practice
Constrained decoding, self-hosted — XGrammar vs Outlines, vLLM/SGLang request shapes, the schema subset, the two ways it lowers accuracyConstrained decoding on self-hosted models Function calling and structured output
Post-processing and the validation ladder — normalise amounts and dates, format checks (IBAN mod-97), evidence match, business rules, calibrated logprob thresholdsStructured extraction from documents(§ Post-processing and validation)Guardrails and output validation
Evaluating extraction — exact match after normalisation, ANLS, TEDS, straight-through rate, missing/wrong/hallucinated taxonomy, the golden set, audits after labels stopEvaluating document extraction

🔴 Fine-tuning — LoRA / QLoRA / PEFT

LoRA mechanics — rank, alpha, target modules, QLoRA’s 4-bit base, adapters at serving timeLoRA and PEFT Quantization SFT and instruction tuning
On a VLM — what to adapt (LLM always, projector often, encoder last), the data format, split by vendor, resolution skew, merge then quantise then re-evaluateFine-tuning VLMs
When not to — prompting and few-shot, retrieval, preprocessing fixes; distillation from a bigger modelFine-tuning vs RAG Distillation and synthetic data Synthetic data and normalization

🔴 Deployment and serving

The engines — vLLM as the default, SGLang, TensorRT-LLM behind Triton; continuous batching, PagedAttention, prefix caching, chunked prefill, speculative decodingBatching and serving engines Model export and runtimes Inference basics
Serving a VLM — max_pixels and images-per-request capped server-side, max-model-len from real pages, FP8 KV cache, prefix caching with stable text before the image, the CPU preprocessing bottleneck, replicas over TPServing VLMs
Quantization — AWQ / GPTQ int4, FP8 on Hopper/Ada, what degrades first; quantise the LLM, keep the encoder in bf16Quantization Serving VLMs(§ Memory, and therefore concurrency)
KV cache arithmetic — bytes per token = 2 × layers × kv_heads × head_dim × dtype; pages in flight from what is left after the weightsKV cache Attention mechanisms (GQA / MLA)
Sizing GPUs against an SLA and a budget — three measured numbers, a worked count, the latency budget as a decode problem, self-host vs API break-even, how to say it to a clientSizing GPUs against an SLA and a budget

Numbers to say without looking: Qwen3-VL-8B ≈ 16 GB bf16 weights; KV ≈ 147 KB/token → ~190 MB per 1,300-token page → ~290 pages in flight in 55 GB, double with FP8 KV. 300k pages/day at a 3× peak is ~10 pages/s → about three H100s at 70% utilisation plus one spare; ~$0.70 per 1,000 pages on-demand versus ~$4 on a hosted API. Single-stream decode of 400 tokens at 80–120 tok/s is 3–5 s — a p95 SLA is met by shortening the output, not adding GPUs. All illustrative; the method is the answer.

🟡 Classical CV, layout detection, OCR

Layout detection and OCR engines — DocLayout-YOLO, Surya, PP-DocLayout; PaddleOCR vs Tesseract vs docTR; table structure; where OCR beats the VLM and the hybrid that shipsLayout detection and OCR engines OCR in CV pipelines
OpenCV preprocessing — deskew, threshold, morphology, line removalOpenCV and preprocessing Contours and polygons
YOLO on documents — stamps, signatures, checkboxes, custom layout classes; NMS; evaluationObject detection CV evaluation CV pipelines and robustness

🟡 Agentic frameworks, RAG, vector databases

LangGraph — state and reducers, routing, persistence, HITL, streaming, production debuggingWhy LangGraph State and reducers Persistence and threads Human in the loop Production and debugging
Agents, honestly — the loop, failure modes, when a workflow beats an agentThe agent loop Agent failure modes Design an agent platform
RAG — chunking, hybrid search and reranking, evaluation, multimodal RAG over pagesWhat is RAG Chunking and retrieval Hybrid search and reranking Agentic RAG and evaluation Design a RAG assistant
Vector databases — pgvector vs Qdrant vs Milvus, HNSW, ingestion and index syncVector databases ANN indexes and HNSW pgvector in production Qdrant in production Ingestion and index sync

🟡 MLOps — MLflow / W&B, CI/CD for models

Tracking and the registry — data hash + commit + config; serve by alias, roll back by aliasExperiment tracking and registry
CI/CD for models — two pipelines, the eval gate against the champion with slice floors, shadow then canary, weights in the image or pulled by aliasCI/CD for models Progressive delivery Deployment strategies
Monitoring after labels stop — review and correction rates, random audits, distribution monitorsMonitoring and drift Evaluating document extraction(§ Once the labels stop)Tracing platforms

🟡 Presales and estimation

Scoping AI work — qualify (data, metric, cost of error), reframe the ask, phases with exits, the pilot that produces numbers, the SOW sections, the demo rulesPresales and scoping AI work Working with clients Estimation and planning
The infra estimate — a range with visible assumptions, narrowed by a two-week pilot on 5,000 documentsSizing GPUs against an SLA and a budget(§ Saying it to a client)ML design framework

The design question to rehearse

“Design invoice extraction for 300k pages a day with a 15-minute turnaround.” Walk it in this order:

Ingest → object storage → queue; content hash as the idempotency key at every stage — Structured extraction, Celery idempotency.
Preprocess: text layer first, render at 200–300 dpi, orientation, page classification, max_pixels chosen on purpose — How a VLM sees, Layout and OCR engines.
Extract with Qwen3-VL-8B on vLLM: nullable schema with evidence quotes, constrained decoding, OCR text alongside the image — Serving VLMs, Constrained decoding.
Validate: format → evidence → business rules → calibrated logprobs; auto-approve or route to review; the review rate is the product metric — Structured extraction (§ Post-processing and validation).
Size it: tokens per page × measured throughput × peak ratio ÷ 0.7 + 1 spare; quantise before adding GPUs; separate batch and online pools — Sizing GPUs.
Evaluate and improve: golden set from the pilot, straight-through rate, corrections into a LoRA fine-tune, merged and quantised and re-evaluated, promoted through the eval gate — Evaluating extraction, Fine-tuning VLMs, CI/CD for models.

Then name the failure modes unprompted — VLM failure modes — and the licence check before any demo — The landscape.

Behavioural — the Company A values as stories

Value	Story to have ready	Notes
Growth	the model family you learned last quarter and what changed in your pipeline because of it	Approaching new technologies · How do you work with AI
Teamwork	an estimate you defended with data, a review that changed a design	Disagreement with the team · Leading technical direction
Ownership	an AI incident you owned — the wrong total that got paid — and the validation rung added after	A failed project or mistake · AI incident response
Openness	telling a client the accuracy number and the review rate before they asked	Working with clients · Presales and scoping
Results	a per-page cost or review rate you moved, with the number	Strongest project · The six signals

Questions to ask them, and the red flags — which VLMs they run today, GPU ownership, who labels, what the review rate isClosing questions and red flags

English: rehearse the design walk-through and the three war stories aloud, in English; the ad’s commercial-experience requirement is checked by asking for numbers, and numbers are easier to say than to improvise.