The chat API surface

What you actually send and receive when you call a model. Everyone writes this code; fewer can explain why it has the shape it does, and the shape is what the follow-up questions are about.

One request, one shape

The Chat Completions API — a list of messages in, one message out — is the de-facto standard. OpenAI defined it, and enough providers copied it that “OpenAI-compatible” is now a feature bullet: Groq, Together, Fireworks, vLLM, Ollama and most gateways accept the same request body.

messages = [
    {"role": "system", "content": "Be brief."},
    {"role": "user", "content": "What is a KV cache?"},
]
resp = client.chat.completions.create(
    model="gpt-5.6", messages=messages,
)
print(resp.choices[0].message.content)

That compatibility is a portability story, not a capability one. The moment you want prompt caching, extended thinking or provider-specific tool semantics you are on native SDKs, because the shared shape only covers the common subset.

The four roles

The system prompt is the operator’s instructions, not the user’s. That distinction is the whole basis of prompt-injection defence: content arriving in a user or tool message is data, however imperative it sounds. See Prompt injection.

The assistant role appearing in your input is what makes multi-turn work — you are replaying the conversation, not continuing a session.

The API is stateless

The provider remembers nothing between calls. Every request carries the entire history, and you are billed for all of it, every turn.

turn 1   send: [sys, u1]                    -> a1
turn 2   send: [sys, u1, a1, u2]            -> a2
turn 3   send: [sys, u1, a1, u2, a2, u3]    -> a3

Two consequences worth stating out loud in an interview:

1. Cost grows quadratically with conversation length if you resend everything. This is the entire reason memory strategies exist — see Memory Strategies for LLM Conversations and Agents.
2. Conversation state is your problem. Session id, transcript storage and what to replay are application concerns.

Stateful server-side conversation APIs do exist, and prompt caching softens the cost curve, but the default mental model is: stateless, resend everything.

Prompt tokens and completion tokens are priced apart

usage on the response reports the real counts — use them rather than estimating, and only fall back to a tokenizer when a provider omits them.

Because output costs multiples of input, “make it cheaper” usually means cap the output, not trim the prompt. A verbose format like JSON with long key names is a recurring, invisible cost.

Streaming changes the felt latency, not the total

Streaming returns tokens as they are generated, so the user sees text at time to first token rather than after the whole completion. Total time is unchanged, sometimes slightly worse.

stream = client.chat.completions.create(
    model="gpt-5.6", messages=messages, stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Gotcha: you cannot validate a structured response you have already shown the user. If output must be schema-valid, either do not stream it or stream to a buffer and render only after it parses. See LLM JSON Validation — Common Interview Questions and Answers.

Streaming also complicates error handling: a failure mid-stream arrives after a 200 OK, so the transport succeeded and the response did not.

Picking a model, and the size question

The model string is the biggest single lever on cost, latency and quality. Parameter counts (8B, 70B, 405B) are a rough proxy: bigger models follow multi-step instructions and choose tools more reliably, and cost more per token.

The 2026 production norm is routing — a small model for classification, extraction and simple replies, a frontier model for reasoning and tool use. Quoting a fixed model for everything reads as not having measured.

Note: as of 2026-08, name a model class rather than a specific version when the point is capability. Version numbers move faster than notes do; see Stack baseline — 2026-2027 for the current table.

Related

• LLM inference fundamentals — what happens inside the call, and sampling
• Model-provider abstraction and LLM resilience — 429s, retries, failover
• Context windows — the window the messages live in
• Function Calling, Tool Use, and Structured Output — the tool role in use