Function calling and structured output

Article.

Verified 2026-08

Verified 2026-08. Provider APIs in this area move; re-check before quoting.

Two things people conflate, and one sentence separates them: function calling is the model asking your code to do something; structured output is the model being forced to answer in a shape. They share a mechanism, and knowing which guarantees you actually get is the difference between parsing confidently and wrapping everything in a retry.

The model does not call your function

Worth being pedantic about, because the wrong mental model produces the wrong error handling.

User:      "What's the weather in Tokyo?"
Model:     emits tokens that decode to
           {"tool": "get_weather",
            "args": {"city": "Tokyo"}}
Your code: get_weather("Tokyo") -> "18C, cloudy"
Model:     "It's 18C and partly cloudy in Tokyo right now."

The model emits a structured request. Nothing executes until your code decides to execute it. That is why every security question here answers itself: the model cannot call an unlisted tool, cannot escalate its own permissions, and cannot bypass a check you put in front of execute() — but it can absolutely produce arguments that are well-formed and wrong.

The loop that carries these calls, and the message-sequence rules it must respect, are in The agent loop.

Defining a tool

The schema is the contract; the description is the prompt.

tools = [{
    "name": "get_weather",
    "description": "Current weather for a city. Use for 'now' questions, "
                   "not forecasts - call get_forecast for future dates.",
    "input_schema": {
        "type": "object",
        "properties": {
            "city": {"type": "string", "description": "City name, e.g. Tokyo"},
            "units": {"type": "string", "enum": ["c", "f"]},
        },
        "required": ["city"],
        "additionalProperties": False,
    },
}]

The description does the tool selection, not the name. That parenthetical telling the model when not to use this tool is worth more than any amount of prompt engineering in the system message — it is read at exactly the moment the decision is made.

Providers differ in wrapper shape and agree on the substance: a name, a description, and a JSON Schema for the arguments.

The mechanism: constrained decoding

Here is what actually separates “the model usually returns JSON” from “the output validates”.

A model samples one token at a time from a probability distribution over the vocabulary. Constrained decoding masks that distribution: at each step, any token that could not appear next in a valid instance of the schema has its probability set to zero before sampling.

schema: {"total": number}

after emitting  {"total":
  allowed next tokens:  0-9  -  "
  masked to zero:       {  [  true  "yes"  ...

The model cannot emit invalid JSON because the invalid continuations are not available to it. That is a structural guarantee, not a strong tendency.

Compare the two ways of asking:

# A request. Usually honoured. Fails at the tail — a stray ```json fence,
# a trailing comma, a chatty preamble, a truncated object at max_tokens.
messages=[{"role": "user", "content": "Return JSON matching {...}"}]

# A guarantee. The sampler cannot produce anything else.
output_config={"format": {"type": "json_schema", "schema": SCHEMA}}

The first is why so much production code has a try: json.loads(...) except: retry around it. With constrained decoding that retry is dead code for malformed JSON — though not for everything, as below.

What the guarantee does not cover

Three limits, each of which surprises someone:

• Shape, not truth. A schema-valid {"total": 4200.0} can still be the wrong total. Constrained decoding removes parse errors, not hallucination.
• A restricted schema subset. Strict modes support a subset of JSON Schema. Typically every property must appear in required — optionality is expressed as a union with null rather than by omission — additionalProperties must be false, and constraints like pattern, minimum or maxItems are commonly unsupported. Validate after parsing anyway, with Pydantic, for the rules the grammar cannot express.
• Truncation still exists. Hit max_tokens mid-object and you get valid- so-far JSON that does not parse. The guarantee is per-token, not per-response.

data = json.loads(response.output_text)   # safe: grammar-constrained
order = Order.model_validate(data)        # still needed: ranges, formats, enums

Structured output, per provider

The current shape on the Anthropic API is output_config.format on the Messages API — note that the older output_format parameter is deprecated, and that the long-standing advice to “force a tool call and read its arguments” is now a legacy workaround, not the recommended path:

# Current: ask for the shape directly.
resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    messages=[{"role": "user", "content": email_text}],
)

# Legacy pattern, still seen everywhere: a tool you never intend to execute.
tools = [{"name": "submit_order", "input_schema": SCHEMA}]
resp = client.messages.create(
    tools=tools, tool_choice={"type": "tool", "name": "submit_order"}, ...
)

The legacy trick worked because tool arguments were always schema-constrained — which is the tell that these are one mechanism wearing two names. If you meet the forced-tool pattern in an existing codebase, it is not wrong, it is dated.

Pin the model version in config. Structured-output behaviour and supported schema subsets change between model versions, and a floating alias turns that into an outage you did not deploy.

Choosing between them

They are not exclusive: a tool’s arguments are structured output. The only question is whether your code does something with the result.

Where it breaks in production

Too many tools. Tool selection degrades as the list grows — descriptions compete, the model picks plausible-adjacent tools, and every schema is in the prompt on every call. Past roughly a dozen, group them behind a router, or use a tool-search mechanism that loads only the relevant schemas.

The enum that drifted. A status enum in a tool schema gets a new value in the database and not in the schema. The model, constrained, can only emit the old values — so it picks the closest one and your handler writes the wrong status. Generate tool schemas from the same source as the domain types, never by hand in a second place.

Nonsense that validates

Schema-valid nonsense. The classic is a date: the schema says string, the model emits "next Tuesday". Constrained decoding is satisfied and your parser is not. Use format hints and enums wherever the value space is closed, and validate ranges after parsing:

class Refund(BaseModel):
    order_id: str
    amount: Decimal = Field(gt=0, le=10_000)   # the grammar cannot say this
    reason: Literal["damaged", "late", "duplicate"]

Argument injection. A tool argument is untrusted input that reached you via a model that read a document, an email or a web page. run_query(sql=...) with model-supplied SQL is the same vulnerability as user-supplied SQL, with an extra step. Treat every argument as hostile, and never expose a tool whose safety depends on the model behaving.

Tool errors that vanish. A failing tool must return its error to the model, not raise into your loop:

try:
    result = await handler(**args)
except ToolError as e:
    result = {"error": str(e), "retryable": e.retryable}

Models recover from “city not found: did you mean Tokyo?” remarkably well. They cannot recover from an exception they never see.

Tool design

• Narrow and well named. get_weather(city), not lookup(query).
• Descriptions that say when not to use it. That is where selection errors come from.
• Enums over free strings wherever the value space is closed — it moves the constraint into the grammar.
• Idempotent where possible, because the model will call the same tool twice; see Idempotency keys.
• Return JSON, not prose. The model is reasoning over the result.

The tools to avoid are the powerful ones: execute_python, query_database, http_request. They are hard for the model to use correctly and impossible to gate meaningfully, because the dangerous action is a string parameter rather than a distinct tool you can put an approval in front of.

The trade not taken

Structured output has a cost people skip: constraining the output can reduce answer quality. Forcing a model to fill a rigid schema on a hard reasoning task gives it nowhere to work — and the fix is to give it somewhere, with a reasoning field before the answer fields, so the schema follows the order the model needs rather than the order you find tidy.

And for a single classification with five outcomes, none of this is needed. Ask for one word, match it case-insensitively, and skip the schema, the grammar compilation and the validation layer entirely.

Related

• The agent loop - the loop these calls drive
• LLM JSON Validation — Common Interview Questions and Answers - validating what comes back
• Model Context Protocol — fundamentals - tools as a protocol across processes
• Constrained decoding on self-hosted models - the same mechanism inside vLLM and SGLang, and how it can lower accuracy
• The Claude Messages API in practice - the full loop on the Anthropic SDK, with streaming, caching and error handling