The chat API surface

What you actually send and receive when you call a model. Everyone writes this code; fewer can explain why it has the shape it does, and the shape is what the follow-up questions are about.

One request, one shape

The Chat Completions API — a list of messages in, one message out — is the compatibility shape. OpenAI defined it, and enough providers copied it that “OpenAI-compatible” is now a feature bullet: Groq, Together, Fireworks, vLLM, Ollama and most gateways accept the same request body.

It is no longer OpenAI’s own primary API: that is the Responses API, with Chat Completions “supported indefinitely” (The OpenAI API in practice); Anthropic’s native shape is the Messages API.

messages = [
    {"role": "system", "content": "Be brief."},
    {"role": "user", "content": "What is a KV cache?"},
]
resp = client.chat.completions.create(
    model=MODEL, messages=messages,       # model id from config, not a literal
)
print(resp.choices[0].message.content)

That compatibility is a portability story, not a capability one. The moment you want prompt caching, extended thinking or provider-specific tool semantics you are on native SDKs, because the shared shape only covers the common subset.

The roles

The system prompt is the operator’s instructions, not the user’s. That distinction is the whole basis of prompt-injection defence: content arriving in a user or tool message is data, however imperative it sounds. See Prompt injection.

The assistant role appearing in your input is what makes multi-turn work — you are replaying the conversation, not continuing a session.

The API is stateless

The provider remembers nothing between calls. Every request carries the entire history, and you are billed for all of it, every turn.

turn 1   send: [sys, u1]                    -> a1
turn 2   send: [sys, u1, a1, u2]            -> a2
turn 3   send: [sys, u1, a1, u2, a2, u3]    -> a3

Two consequences worth stating out loud in an interview:

1. Cost grows quadratically with conversation length if you resend everything. This is the entire reason memory strategies exist — see Memory Strategies for LLM Conversations and Agents.
2. Conversation state is your problem. Session id, transcript storage and what to replay are application concerns.

Stateful server-side conversation APIs do exist: OpenAI’s Responses API can store responses (store, previous_response_id) or keep a Conversation object. They save sending the payload, not the bill: “all previous input tokens for responses in the chain are billed as input tokens”. Only prompt caching softens the cost curve, so the mental model stays: every turn pays for the whole history.

Prompt tokens and completion tokens are priced apart

usage on the response reports the real counts — use them rather than estimating, and only fall back to a tokenizer when a provider omits them.

Because output costs multiples of input, “make it cheaper” usually means cap the output, not trim the prompt. A verbose format like JSON with long key names is a recurring, invisible cost.

Streaming changes the felt latency, not the total

Streaming returns tokens as they are generated, so the user sees text at time to first token rather than after the whole completion. Total time is unchanged, sometimes slightly worse.

stream = client.chat.completions.create(
    model=MODEL, messages=messages, stream=True,
    stream_options={"include_usage": True},   # otherwise no usage at all
)
for chunk in stream:
    if chunk.choices:                         # the final usage chunk has none
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    if chunk.usage:
        record_usage(chunk.usage)

The usage chunk arrives last, with an empty choices list, so a loop that reads chunk.choices[0] unconditionally crashes on exactly the chunk that carries the bill; and a stream cut off early never sends it.

Gotcha: you cannot validate a structured response you have already shown the user. If output must be schema-valid, either do not stream it or stream to a buffer and render only after it parses. See LLM JSON Validation — Common Interview Questions and Answers.

Streaming also complicates error handling: a failure mid-stream arrives after a 200 OK, so the transport succeeded and the response did not.

Picking a model, and the size question

The model string is the biggest single lever on cost, latency and quality. Parameter counts (8B, 70B, 405B) are a rough proxy: bigger models follow multi-step instructions and choose tools more reliably, and cost more per token.

The 2026 production norm is routing — a small model for classification, extraction and simple replies, a frontier model for reasoning and tool use. Quoting a fixed model for everything reads as not having measured.

Note: as of 2026-08, name a model class rather than a specific version when the point is capability. Version numbers move faster than notes do; see Stack baseline — 2026-2027 for the current table.

Related

• The Claude Messages API in practice — the same shape on Anthropic’s API, with the tool loop and caching
• LLM inference fundamentals — what happens inside the call, and sampling
• Model-provider abstraction and LLM resilience — 429s, retries, failover
• Context windows — the window the messages live in
• Function calling and structured output — the tool role in use