Inference and serving
The operational layer — what determines latency, throughput and cost.
| # | File | Covers |
|---|---|---|
| 01 | LLM inference fundamentals | prefill vs decode, metrics, sampling, streaming, cost model, reliability |
| 02 | Batching and serving engines | continuous batching, PagedAttention, prefix caching, speculative decoding, vLLM |
| 03 | Model-provider abstraction and LLM resilience | one interface over many providers, retries, fallback, caching |
| 04 | The chat API surface | the request shape, message roles, statelessness, token pricing |
| 05 | Which model for which task | which model for which task, escalation rate, cost levers |
| 06 | The LLM gateway and caching | the gateway layer, exact/prefix/semantic caching, invalidation |
| 07 | Azure and Vertex | Azure OpenAI, AI Search, Foundry; Vertex; managed RAG or build it |
| 08 | Model export and inference runtimes | ONNX, TorchScript, the runtime, and the integration contract |
| 09 | AWS Bedrock and AgentCore | AWS managed inference |
| 10 | Determinism and reproducible LLM tests | why temperature 0 is not deterministic, and how to test anyway |
| 11 | The Claude Messages API in practice | the Anthropic Messages API: shape, tool loop, structured outputs, streaming, caching, batches, errors |
| 12 | The OpenAI API in practice | the OpenAI API: Responses vs Chat Completions, structured outputs, tools, streaming events, usage and caching, Batch/Flex, rate limits |
The cost levers, in order of return
- Route by difficulty — cheap model for easy requests, frontier only when needed.
- Prefix caching — stable content first in the prompt; ~30% throughput gain when requests share a system prompt.
- FP8 KV cache — halves cache memory, doubles concurrency, negligible quality cost.
- Cap
max_tokens— output tokens are the expensive half. - Batch offline work — asynchronous batch APIs are heavily discounted.
Latency and throughput trade off directly. There is no setting that maximises both; you optimise one under a constraint on the other.