The OpenAI API in practice

How do you call the OpenAI API correctly in production in 2026? The answer starts with which API: Responses is OpenAI’s primary interface (March 2025), Chat Completions is the older shape that it supports indefinitely and that most other providers copy, and the Assistants API was shut down on 2026-08-26. Code that creates threads and runs no longer works.

After that come the details that separate a demo from a service: structured outputs, the tool loop, streaming events, how usage and caching are billed, Batch and Flex, rate limits, and retries that do not double the bill.

The chat API surface is the provider-neutral view; The Claude Messages API in practice is the Anthropic equivalent of this note. Python SDK (openai 3.x); model names and prices change monthly, so code reads the model from config, and prices below are the list prices as of October 2026.

Responses or Chat Completions

from openai import OpenAI

client = OpenAI()                         # OPENAI_API_KEY from the environment
MODEL = settings.openai_model             # never a literal scattered in code

resp = client.responses.create(
    model=MODEL,
    instructions="You are a support assistant for Acme. Answer from the docs.",
    input="How do I rotate an API key?",
    max_output_tokens=4000,
)
print(resp.output_text)                   # helper: joins the text output items
print(resp.status, resp.usage.input_tokens, resp.usage.output_tokens)

On the wire

What actually differs:

• Output is a list of items, not one message: message, function_call, reasoning, web_search_call and so on. output_text is a convenience; code that handles tools iterates resp.output.
• Reasoning is carried across tool calls. On reasoning models, the reasoning items from one turn feed the next, which OpenAI credits for a quality gain and for 40-80% better cache utilisation than Chat Completions in its internal tests.
• State can live on the server: store defaults to true (kept for at least 30 days), and the next call can pass previous_response_id, or a conversation from the Conversations API (the two are mutually exclusive).

Chat Completions stays the right choice when portability matters: it is the shape vLLM, Ollama, Azure, Gemini’s compatibility endpoint and most gateways accept. For a new agent on OpenAI, Responses.

What server-side state does not save

“Even when using previous_response_id, all previous input tokens for responses in the chain are billed as input tokens in the API.”

previous_response_id saves you sending the payload and keeps reasoning state; it does not make history free. Only prompt caching discounts those tokens.

And store=False (required by some compliance setups) means there is nothing to chain from: you resend the items yourself, and on reasoning models pass back the encrypted reasoning content (request it with include=["reasoning.encrypted_content"] where it is not returned by default).

Structured outputs

from pydantic import BaseModel

class Ticket(BaseModel):
    category: str
    priority: int
    summary: str

resp = client.responses.parse(
    model=MODEL,
    input=email_text,
    text_format=Ticket,                   # Chat Completions: response_format=Ticket
)
ticket = resp.output_parsed               # a Ticket, or None on refusal

With a strict JSON Schema the output is constrained to the schema, not just asked to follow it. The strict subset has rules that surprise people:

• Every property must be listed in required. An optional field is expressed as a union with null, not by leaving it out.
• additionalProperties: false on every object.
• A new schema has a one-off compilation delay on its first request; later requests with the same schema do not pay it.
• A refusal is not JSON: the model may decline, and the output then carries a refusal instead of parsed content. Check before using it.

The older JSON mode ({"type": "json_object"}) guarantees only that the output parses, with no schema, and errors unless the word “JSON” appears in the context. Grammar constraints guarantee shape, not truth: validate ranges and cross-field rules after parsing (LLM JSON Validation — Common Interview Questions and Answers).

The tool loop

tools = [{
    "type": "function",                   # Responses: flat shape
    "name": "get_order",
    "description": "Look up an order by its id.",
    "parameters": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
        "additionalProperties": False,
    },
    "strict": True,
}]

items = [{"role": "user", "content": "Where is order A-1042?"}]
while True:
    resp = client.responses.create(model=MODEL, input=items, tools=tools)
    items += resp.output                  # keep every item, reasoning included
    calls = [i for i in resp.output if i.type == "function_call"]
    if not calls:
        break
    for call in calls:
        result = run_tool(call.name, json.loads(call.arguments))
        items.append({"type": "function_call_output",
                      "call_id": call.call_id, "output": json.dumps(result)})
print(resp.output_text)

What differs between the two APIs

• Chat Completions nests it as {"type": "function", "function": {...}} and returns message.tool_calls; answers go back as role: "tool" messages with tool_call_id.
• Strict by default, with a silent fallback. In Responses, omitting strict attempts strict mode, and if the schema cannot be made compatible, it falls back to non-strict best-effort calling and reports strict: false on the resolved tool. Set strict explicitly and alert on that downgrade; Chat Completions defaults to non-strict.
• tool_choice: auto, required (some tool, any of them), none, a named function, or allowed_tools to restrict the set per turn. required forces a call; it does not make arguments valid.
• Built-in tools run on OpenAI’s side, including remote MCP servers. require_approval on an MCP tool is the boundary between a prompt injection in a fetched page and an action taken on your behalf.

Streaming: two different wire formats

stream = client.responses.create(model=MODEL, input=prompt, stream=True)
for event in stream:
    if event.type == "response.output_text.delta":
        send_to_client(event.delta)
    elif event.type == "response.completed":
        record_usage(event.response.usage)          # only here
    elif event.type in ("response.failed", "response.incomplete", "error"):
        handle_failure(event)

• Responses streams named events (response.created, response.output_item.added, response.output_text.delta, response.function_call_arguments.delta, response.completed), each with a sequence_number.
• Chat Completions streams unnamed chunks of choices[].delta, ending with data: [DONE]. Usage is included only with stream_options={"include_usage": True}, on an extra final chunk whose choices is empty, so a loop that reads chunk.choices[0] raises IndexError on exactly the chunk that carries the bill:

for chunk in client.chat.completions.create(
        model=MODEL, messages=msgs, stream=True,
        stream_options={"include_usage": True}):
    if chunk.choices:
        send_to_client(chunk.choices[0].delta.content or "")
    if chunk.usage:
        record_usage(chunk.usage)

In both APIs, usage arrives only at the end. A stream cut off by the client or the network has billed tokens and reports none; reconcile from the organisation usage API. Deltas also carry an obfuscation padding field by default, a mitigation against inferring content from packet sizes; turning it off saves bandwidth and gives that up. Re-streaming to a browser is in Server-Sent Events (SSE).

Long runs: background mode

background=True starts a response that runs without holding the HTTP connection; poll client.responses.retrieve(id) or stream it, and after a dropped stream resume with starting_after=<last sequence_number>. Retrieve by id after a failure instead of creating it again, which would be a second billed generation.

Usage, caching and what a call costs

The usage fields (Responses names; Chat Completions uses prompt_ and completion_ prefixes):

Two consequences. Reasoning tokens are billed as output and count against max_output_tokens, so a low cap on a reasoning model can end with status incomplete (reason max_output_tokens) and no visible text at all. And caching is no longer free: from GPT-5.6 on, cache writes cost 1.25x the input price and reads 0.1x (0.05x on gpt-6.1-sol); older models cached automatically with free writes.

# gpt-6.1-sol list prices, USD per 1M tokens (October 2026)
PRICE = {"in": 2.00, "cache_write": 2.50, "cache_read": 0.10, "out": 10.00}

def cost_usd(u) -> float:
    d = u.input_tokens_details
    read, write = d.cached_tokens or 0, getattr(d, "cache_write_tokens", 0) or 0
    plain = u.input_tokens - read - write
    return (plain * PRICE["in"] + write * PRICE["cache_write"]
            + read * PRICE["cache_read"] + u.output_tokens * PRICE["out"]) / 1e6

Caching works on prefixes of at least 1,024 tokens, so stable content (instructions, tool definitions, documents) goes first and per-request data last. prompt_cache_key groups requests that share a prefix; from GPT-5.6 it only separates cache accounting. List prices for scale: gpt-6-astra $10/$50, gpt-6.1-sol $2/$10, gpt-6-luna $0.10/$0.50 per million input and output tokens, output 5x input across the current family.

Batch and Flex: half price for waiting

• Batch: a JSONL file of {custom_id, method, url, body} lines, uploaded with purpose="batch", then client.batches.create( input_file_id=..., endpoint="/v1/responses", completion_window="24h"). 50% off; up to 50,000 requests or 200 MB per batch; status goes validating → in_progress → finalizing → completed (or failed, expired, cancelled).
• Batch results come back unordered: join on custom_id. An expired batch still bills the requests that completed, and batches draw on a separate queue limit, not your per-minute token limit.
• Flex (service_tier="flex") is the synchronous cheap tier, priced like Batch, for work that can tolerate slower responses and being turned away when capacity is short. Priority tiers are the opposite trade.

Rate limits and retries

Limits are per model: requests and tokens per minute and per day, plus project-level limits. Two details explain most surprise 429s:

• Admission counts the larger of max_output_tokens and the estimate. A generous cap reserves throughput the call never uses; at 40% of your real TPM you can be throttled.
• Failed requests count too. A retry storm consumes the limit it is waiting for.

Read the headers (x-ratelimit-remaining-tokens, x-ratelimit-reset-tokens, retry-after) through with_raw_response, and branch on the error code:

client = OpenAI(max_retries=2, timeout=60.0)      # explicit, not 600 s

try:
    raw = client.responses.with_raw_response.create(model=MODEL, input=prompt)
    resp = raw.parse()
    metrics.gauge("openai_tokens_remaining",
                  int(raw.headers.get("x-ratelimit-remaining-tokens", 0)))
except openai.RateLimitError as e:
    if e.code == "insufficient_quota":            # billing stop: no retry helps
        page_oncall("OpenAI quota exhausted")
    raise
except openai.APIStatusError as e:
    log.error("openai %s %s req=%s", e.status_code, e.code, e.request_id)
    raise

Retry once, in one layer

• The SDK already retries connection errors, 408, 409, 429 and 5xx twice, with 0.5-8 s backoff, honouring retry-after. A tenacity loop of four attempts around it is up to twelve HTTP attempts per call, all counted against the limit. Configure one layer.
• The default timeout is 600 s. Set one that fits under every proxy on the path (an ALB’s 60 s idle timeout, your own read timeout); a timed-out attempt that is retried may be a second billed generation.
• There is no idempotency key for Responses or Chat Completions. Deduplicate in your own layer (request hash to stored response id), and for background responses retrieve by id.
• The 3.x SDK moved its HTTP layer from httpx to HTTPX2; custom http_client and transport objects from 2.x code need migrating.

Where it breaks in production

• Assistants-era code: threads, runs and assistants endpoints return errors since 2026-08-26. Rewrite around Responses and Conversations.
• Streams that log zero usage: include_usage missing, or the loop crashed on the empty-choices chunk, or the stream was cut before the terminal event.
• Quota 429s retried forever: insufficient_quota treated as back-pressure.
• Reasoning starved by max_output_tokens: status incomplete, empty output, full charge for the reasoning.
• store=False and chaining: previous_response_not_found because nothing was stored.
• Twelve attempts per call: SDK retries nested inside an application retry loop during an outage.
• Parameters a model rejects: temperature, top_p and stop are not accepted by several reasoning models; a request copied from an older model returns 400.

Related

• The chat API surface - the provider-neutral shape
• The Claude Messages API in practice - the Anthropic equivalent
• Model-provider abstraction and LLM resilience - fallbacks and breakers around the call
• The LLM gateway and caching - gateways and application-level caching
• Function calling and structured output - constrained decoding and tool design
• Worked Design — LLM usage metering - recording usage and cost per tenant