AI & ML / Vision language models / 05_constrained_decoding_self_hosted.md

Constrained decoding on self-hosted models

Updated 4 interview angles 6 min read source
On this page7
  1. What the engine does per token
  2. Asking for it
  3. The schema subset
  4. The two ways it lowers accuracy
  5. Where it breaks in production
  6. Related
  7. Interview angle

Constrained decoding on self-hosted models

On a hosted API “structured output” is a checkbox. On a model you serve yourself it is a feature of the engine, with a backend to choose, a schema subset to respect, a compile step to cache and two ways to quietly lower accuracy. The mechanism is in Function calling and structured output; this note is the operational half.

What the engine does per token

python 
# The idea, stripped of the engineering: before sampling each token,
# zero out every token that cannot continue a valid output.
def allowed(prefix: str, vocab: list[str], pattern) -> set[int]:
    return {i for i, tok in enumerate(vocab)
            if pattern.partial_match(prefix + tok)}   # can it still match?

logits[[i for i in range(V) if i not in allowed(out, vocab, pat)]] = -inf

Doing that literally is far too slow — 150,000 vocabulary entries per step — so the backends precompute. Regex and JSON Schema become an automaton over the tokenizer’s vocabulary: for every automaton state, the set of tokens that keep it valid is computed once and stored as a bitmask, and each decoding step is a table lookup plus a mask applied to the logits before sampling. The compile happens once per schema and is cached by hash.

Two families of backend:

Backend Approach
Outlines regex/JSON → finite-state machine, mask per state
XGrammar context-free grammar, pushdown automaton, cached masks
llguidance Earley parser, lazy token-level masks

XGrammar is the default in vLLM and SGLang because a pushdown automaton handles recursive schemas (nested objects, arrays of arrays) that a pure FSM cannot, and its mask computation overlaps with the GPU so the per-token overhead is close to zero.

SGLang adds jump-forward decoding: when the grammar allows exactly one continuation — the next key name, a closing brace — those tokens are appended without a forward pass, which makes fixed-key JSON measurably faster than free text.

Asking for it

python 
from openai import OpenAI

client = OpenAI(base_url="http://vllm:8000/v1", api_key="-")

resp = client.chat.completions.create(
    model="qwen3-vl-8b",
    messages=messages,
    response_format={"type": "json_schema",
                     "json_schema": {"name": "invoice", "schema": SCHEMA}},
    # vLLM-specific alternatives, via extra_body:
    #   {"structured_outputs": {"regex": r"\d{4}-\d{2}-\d{2}"}}
    #   {"structured_outputs": {"choice": ["approve", "reject"]}}
    #   {"structured_outputs": {"grammar": EBNF}}
    # (older releases spell these guided_json / guided_regex / guided_choice)
)

vLLM serves the OpenAI shape — response_format with a JSON Schema — and its own structured_outputs field for regex, choice and EBNF grammars; the guided_* names are the pre-2025 spelling and still appear in most blog posts. SGLang takes json_schema, regex and ebnf in the sampling parameters. The backend is a server flag; the client does not see it.

With a reasoning model the constraint has to start after the thinking block, or the grammar forbids the model from thinking. Both engines have a reasoning parser for exactly this; enable it or the Thinking variant produces JSON with no reasoning and worse answers than the Instruct one.

The schema subset

Every backend implements a subset of JSON Schema, and the subset differs. The usual casualties: numeric minimum/maximum, format (dates, emails), complex patterns, deep $ref recursion, oneOf with overlapping branches. What is reliably supported: types, required, enum, const, anyOf, additionalProperties: false, string and array length limits.

Two rules follow. Compile the schema in CI against the pinned engine version — a schema the backend rejects at request time is an outage, not a bug report. And validate with Pydantic after parsing for every rule the grammar cannot express; the grammar guarantees shape, Pydantic checks ranges, formats and the cross-field rules (Structured extraction from documents).

The two ways it lowers accuracy

Forced commitment. The grammar makes the first token {, the second the first key, and the value right after. A model that would have written “the total appears twice; the second is the corrected one” before answering never gets to. On easy pages nothing changes; on hard pages accuracy drops measurably.

Fixes: order the schema so a free-text analysis field comes first, or use a Thinking variant with the reasoning kept outside the grammar, or run two calls — free text, then extraction from the text.

Forced choice. With an enum, the sampler picks the highest-probability allowed token, and when the model’s intended answer is not in the enum the pick is whichever option shares a prefix with what it wanted to say — often the first alphabetically. Add an "other" member and a free-text field beside it, and treat "other" as a review signal.

Neither is an argument against constraining; unconstrained JSON on an open-weight model fails to parse a few percent of the time, and that failure mode is worse. It is an argument for measuring field accuracy with and without the grammar on your golden set before choosing the schema layout.

Where it breaks in production

  • Compile latency on first request. A large schema compiles in seconds on the first call and is cached after; a deploy that rotates pods sees a latency spike per pod. Warm the cache at startup with the schemas you serve.
  • Whitespace freedom. Some grammars allow arbitrary whitespace between tokens, and a model in a bad state emits kilobytes of it. Disable whitespace flexibility where the backend offers it, and keep max_tokens tight.
  • Truncation. The grammar is enforced per token; max_tokens still cuts the object in half. Size the cap from the schema’s largest legal output.
  • Tokenizer mismatch after a model swap. The mask cache is per tokenizer; a fine-tune with added tokens or a different chat template invalidates it, and a stale cache produces impossible outputs.
  • Batch slowdown from one hard grammar. In a shared batch, one request with an expensive grammar can hold the step for everyone. Watch per-step latency when adding a new schema, and prefer simpler shapes.

Interview angle 4

  • “How does constrained decoding work in vLLM?” - the schema is compiled once into an automaton over the tokenizer’s vocabulary - a pushdown automaton in XGrammar so nesting works - with the allowed-token set per state stored as a bitmask. Each step masks the logits before sampling; the compile is cached by schema hash and the per-token cost is overlapped with the GPU. SGLang adds jump-forward: tokens the grammar fully determines are appended without a forward pass.
  • “Does it make the output correct?” - it makes it parse. Shape, not truth: the model can still invent a value, and the grammar can only express a subset of JSON Schema, so Pydantic runs after for ranges, formats and cross-field rules. Follow-up: can it hurt? Yes - forcing the answer before any reasoning lowers accuracy on hard inputs; put a free-text field first or keep reasoning outside the grammar.
  • “Outlines or XGrammar?” - XGrammar for JSON Schema in production: CFG handles recursion, near-zero overhead, default in both engines. Outlines when you want the library standalone with plain transformers, or a regex is the natural spec.
  • “What goes wrong operationally?” - first-request compile spikes after a deploy, whitespace loops, max_tokens truncating a valid-so-far object, and a stale mask cache after swapping in a fine-tune with a different tokenizer. Warm the cache, tighten the cap, and compile every schema in CI against the pinned engine.
Previous
Structured extraction from documents