HTTP semantics and caching

Article.

HTTP semantics are the parts that stay the same across HTTP/1.1, HTTP/2 and HTTP/3: methods, status codes, header fields, content negotiation, conditional requests and caching. RFC 9110 defines them, RFC 9111 defines caching and RFC 9112 the HTTP/1.1 wire format; all three appeared in June 2022 and replaced RFC 7230-7235. Version differences are in HTTP versions: 1.1, 2 and 3, message basics in HTTP.

Behind “which methods are idempotent?” sits a better question: what each method licenses software you do not control to do with a request. Crawlers, retry loops, proxies and caches act on these properties without asking you.

Safe, idempotent, cacheable

RFC 9110 section 9.2 defines three properties, each narrower than the folklore:

• Safe: the client does not request, and does not expect, a state change on the server. Logs, metrics and an audit row are allowed; what counts is that the client did not ask for the side effect and cannot be held accountable for it.
• Idempotent: the intended effect of N identical requests equals the effect of one. DELETE answering 204 and then 404 is idempotent: the state is the same, and the response is allowed to differ.
• Cacheable: the method’s definition allows a response to be stored and reused. Necessary, not sufficient: RFC 9111 still checks the status code, Cache-Control and Authorization before anything is stored.

The retry licence is narrow. A client may repeat an idempotent request “if a communication failure occurs before the client is able to read the server’s response”; it SHOULD NOT repeat a non-idempotent one unless it knows the repeat is harmless, which in practice means an idempotency key (Idempotency keys).

Every method

• POST and PATCH are cacheable on paper - only with explicit freshness plus a Content-Location equal to the target URI, and RFC 9110 admits that “the overwhelming majority of cache implementations only support GET and HEAD.”
• OPTIONS responses are never cacheable. A CORS preflight is cached by Access-Control-Max-Age instead: 5 s by default, capped at 7200 s in Chromium and 86400 s in Firefox (CORS — The Deep Dive).
• PATCH is not idempotent by definition, though a given patch can be: “set status to paid” is, “append to the list” is not (PUT vs PATCH: which writes are safe to retry).
• A body on GET, HEAD or DELETE has no defined semantics, and RFC 9110 warns that servers may reject one as a possible smuggling attempt.
• TRACE echoes the request back, which once let scripts read HttpOnly cookies (cross-site tracing); browsers may not send it, and servers should disable it. CONNECT host:port is how HTTPS crosses a forward proxy.
• An unrecognised method gets 501; a recognised one the resource refuses gets 405, which MUST list the allowed methods in Allow.

QUERY: a safe read with a body

QUERY became RFC 10008 in June 2026. It fills the gap between GET, which cannot carry a body, and POST, which can but promises neither safety nor idempotency - the old workaround for search endpoints with large filters:

QUERY /orders HTTP/1.1
Host: api.example.com
Content-Type: application/json

{"status": "open", "customer_id": {"in": [17, 42, 99]}, "limit": 50}

It is safe and idempotent, so clients may retry it and caches may store it, with the request body in the cache key. As of 2026-10 it is new: check that every hop - CDN, proxy, client library - passes the method through before relying on it.

Who retries for you

Retries happen in code you did not write, and the safe/idempotent properties are what they key on:

Read from the installed libraries on 2026-10-15:

from urllib3.util import Retry
sorted(Retry.DEFAULT_ALLOWED_METHODS)
requests.adapters.HTTPAdapter().max_retries
httpx.HTTPTransport()._pool._retries

urllib3 2.8.0    ['DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE']
requests 2.34.2  Retry(total=0, connect=None, read=False, redirect=None, status=None)
httpx 0.28.1     0

So POST and PATCH are never retried for you, and with requests and httpx nothing is unless you configure it. A retry policy for outbound calls is in Writing a client for an external API.

Keep-alive is per hop, and when a load balancer reuses an idle connection the app has just closed, the request meets a reset and the user gets a 502. The app’s idle timeout must outlive the load balancer’s (uvicorn --timeout-keep-alive 75 behind an ALB’s 60 s); the full story is in What happens when you enter a URL.

Content negotiation

The request states preferences, the server picks a representation and labels it, and Vary records which request fields it consulted:

GET /reports/42 HTTP/1.1
Accept: application/json, text/csv;q=0.5
Accept-Encoding: zstd, br, gzip

HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: br
Vary: Accept, Accept-Encoding

With nothing acceptable the server may answer 406 or send its default; a request body in a format it cannot read gets 415. Content-Encoding belongs to the representation and travels end to end; Transfer-Encoding is per-hop framing.

Conditional requests

A conditional request carries a validator from an earlier response and asks the server to act only if the resource is, or is not, still in that state. Cache revalidation and lost-update prevention are the same mechanism.

Validators

• ETag: "v7" is strong: it changes on any byte-level change to the representation, content coding included.
• ETag: W/"v7" is weak: it promises only semantic equivalence.
• Last-Modified is an HTTP date with one-second resolution, so two writes in one second share a value; treat it as weak. A version counter or a content hash makes a good strong ETag.

If-Match must use strong comparison; If-None-Match uses weak.

Preconditions

They are evaluated in a fixed order (RFC 9110 section 13.2.2) and only after the normal checks, so a request that would get 404 or a redirect gets that, never 412. 428 Precondition Required (RFC 6585) lets a server make If-Match mandatory.

The lost-update fix is “write only if the version I read is still current”, and the comparison and the write must be one statement, or two writers both pass the check:

UPDATE docs SET body = $1, version = version + 1
WHERE id = $2 AND version = $3   -- the version from the client's If-Match
RETURNING version;                -- no row back: answer 412

Gotcha: since 1.7.3, nginx turns a strong ETag weak whenever gzip or another filter rewrites the response. A client that echoes it in If-Match then gets 412 on every update. Compress in the app with a strong tag per encoding, or know that only the opaque part is being compared.

The full PUT and PATCH contracts, with If-Match in a FastAPI handler, are in PUT vs PATCH: which writes are safe to retry.

Caching

Private and shared caches

A private cache serves one user: the browser’s. A shared cache serves many: a CDN, nginx proxy_cache, Varnish, a corporate proxy. One response can pass through several:

browser ──▶ CDN edge ──▶ CDN shield ──▶ nginx ──▶ app
private     shared       shared         shared

The cache key is the method plus the target URI, extended by the request fields the response names in Vary. Browsers also add the top-level site (“double keying”), so one site cannot learn from load times what you visited elsewhere.

What may be stored

RFC 9111 section 3 lets a cache store a response only when all of these hold:

1. The method is understood (in practice GET or HEAD) and the status is final.
2. There is no no-store.
3. For a shared cache: no private, and if the request carried Authorization, the response says public, s-maxage or must-revalidate.
4. There is explicit freshness (max-age, s-maxage, Expires), public or private, or a heuristically cacheable status such as 200, 301, 404 or 410.

Rule 4 is the one people miss: a 200 with no caching headers at all is storable and gets a heuristic lifetime. Rule 3 is why a CDN does not hand one user’s authenticated response to another by default.

Gotcha: Set-Cookie does not make a response uncacheable. nginx’s proxy_cache happens to skip such responses; other caches need not. Mark personalised responses private or no-store.

Freshness and revalidation

A stored response is fresh while its age is below its freshness lifetime, and a fresh response is served without asking the origin. The lifetime is the first of these that applies (RFC 9111 section 4.2.1):

1. s-maxage, for shared caches only.
2. max-age - which overrides Expires.
3. Expires minus Date.
4. A heuristic, typically 10% of the time since Last-Modified.

A stale copy is revalidated with If-None-Match or If-Modified-Since, and a 304 refreshes it without resending the body. A cache may serve a stale copy without asking only when disconnected or explicitly allowed, and never against no-cache or must-revalidate.

The directives

The four that get confused:

Vary

Vary names the request fields that shaped a response, and caches add them to the key:

• Vary: Accept-Encoding wherever responses may be compressed, or a cache hands gzip to a client that cannot read it.
• Vary: Origin wherever CORS headers echo the request’s Origin.
• Vary: Cookie or Vary: User-Agent splits the key per user or per browser build, and the hit rate falls toward zero.
• Vary: Authorization is unnecessary: shared caches already refuse to reuse authenticated responses unless told otherwise (rule 3 above).

Invalidation and what the cache did

A successful unsafe request (POST, PUT, DELETE) that passes through a cache makes it invalidate the target URI (RFC 9111 section 4.4). Only caches on that request’s path hear about it; every other browser and CDN node keeps its copy until it goes stale - hence versioned asset URLs, and purge APIs or short lifetimes for everything else (CDN (Content Delivery Network)).

To see what happened, read Age (seconds since the origin generated or last validated the response) and Cache-Status (RFC 9211), one entry per cache:

Age: 97
Cache-Status: "Edge"; fwd=stale; fwd-status=200; stored

CDN-Cache-Control (RFC 9213) is a Cache-Control aimed only at CDNs, which other caches ignore - the standard way to let the edge cache for an hour while browsers revalidate every time.

Recipes

Browser-side layers are in HTTP Caching Headers — Cache-Control, ETag, Vary, nginx’s proxy_cache in Nginx Caching, and stampede control in Cache Stampede and Mitigation Patterns.

HTTP/1.1 on the wire

Where a message ends

HTTP/1.1 sends messages back to back on a persistent connection, so each one must say where it ends. RFC 9112 section 6.3 decides, in order:

1. A response to HEAD, and any 1xx, 204 or 304 response, has no body.
2. Transfer-Encoding and Content-Length together: Transfer-Encoding wins, and the message “ought to be handled as an error”.
3. Transfer-Encoding: chunked: read chunks until the zero-size one.
4. A valid Content-Length: exactly that many bytes.
5. Otherwise a request has no body, and a response runs until the server closes the connection.

HTTP/2 and HTTP/3 frame messages themselves, with DATA frames and an end-of-stream flag, so this whole class of problem lives where a proxy translates between versions or speaks HTTP/1.1 to the back end.

Request smuggling

Smuggling is a disagreement about where a request ends between a front end and a back end that share a keep-alive connection. In the classic CL.TE case the front end frames by Content-Length and the back end by Transfer-Encoding:

The front end applied its auth and routing rules to request 1 only; the back end answers a second request nobody checked, on the same connection. TE.CL reverses the roles, and HTTP/2 front ends that downgrade to HTTP/1.1 create H2.CL and H2.TE variants. Defences, strongest first:

1. Speak HTTP/2 to the back end, so lengths come from frames - the conclusion of PortSwigger’s “HTTP/1.1 must die” research (August 2025).
2. Reject ambiguous framing at the edge: both length headers, a body on a method that needs none. nginx has rejected both headers together since 1.21.1; AWS ALB’s default desync mode blocks the severe cases.
3. Run a strict parser at the origin (uvicorn with httptools rejects the combination with 400; the pure-Python h11 parser is more lenient).
4. Detect it by comparing per-connection request counts in front and back logs.

Hop-by-hop headers

Some fields describe one connection, not the message: Connection, Keep-Alive, TE, Transfer-Encoding, Upgrade, and any field named in Connection. A proxy strips them before forwarding. So keep-alive is negotiated per hop, Upgrade does not survive a proxy unless it is re-added for WebSocket routes (Nginx WebSocket Proxying), and HTTP/2 forbids these fields outright.

Status codes worth knowing here

Classes and the confusable pairs live in Status codes. What RFC 9110 adds that interviews probe:

Error bodies follow RFC 9457 (Problem Details), which obsoleted RFC 7807.

Where it goes wrong

• State change on GET. Mail-security scanners and chat link previews fetch the URLs in messages, so a one-time login or unsubscribe link that acts on GET is used up before the user clicks. Show a page whose button POSTs.
• Web cache deception: a CDN rule caches /account/settings/x.css by its extension while the origin ignores the suffix and serves the private page, now public. Cache by Cache-Control, never by extension.
• Web cache poisoning: a header the cache does not key on, such as X-Forwarded-Host, is reflected into a cached page, and every later visitor gets the attacker’s version. Key on what you reflect, or stop reflecting it.
• A 301 without Cache-Control is kept by browsers indefinitely, so a mistaken permanent redirect outlives its fix. Use 302 or 307 until sure.

The trade not taken: no-store everywhere

The “safe” policy after a caching incident is Cache-Control: no-store on every response. It does prevent the private-page leaks above, and it throws away everything caching was for: no revalidation, so a client that already has the current version downloads it again instead of getting a 304; no CDN offload for assets and public pages; and a slower site for every visitor to avoid a bug in a few routes.

The policy that holds up is per route, stated explicitly:

no-store belongs in the last row only. A framework default of private, no-cache for anything behind a session cookie, with routes opting into public caching, is the safe direction to fail in.

Related

• HTTP - message anatomy, URLs, cookies
• HTTP versions: 1.1, 2 and 3 - what HTTP/2 and HTTP/3 change on the wire
• Status codes - status classes and the confusable pairs
• Idempotency - making POST retryable
• PUT vs PATCH: which writes are safe to retry - the PUT and PATCH contracts
• Idempotency keys - idempotency keys without the race
• What happens when you enter a URL - the whole request path