Idempotency keys

Article.

A client sends POST /charges, the response never arrives, and the client retries. Did the first one succeed? Neither side knows. Without idempotency you charge the card twice; with it, the second request returns the first response and does nothing else.

The pattern is easy to describe and easy to implement with a race in it. The interesting part is the concurrency, and the naive implementation — including the one most articles show — does not survive two requests arriving at once.

The contract

The client generates a key per logical operation and sends it as a header:

POST /v1/charges
Idempotency-Key: a3f7b9d1-1c8c-4f8c-9e6e-1234abcd5678

{ "amount": 1000, "currency": "usd", "customer": "cus_42" }

The server stores key → response. Replaying the key returns the stored response verbatim — same status code, same body — without re-executing.

Two things the client owes, which are half the pattern and are usually missing from home-grown clients:

• One key per logical operation, persisted before the first attempt. A key generated fresh on each retry is not an idempotency key, it is a new request.
• Retry on network errors and 5xx, and on exactly four 4xx codes: 409 (this key is in progress), 408, 425 Too Early, and 429 after Retry-After. Any other 4xx (a 422 above all) means the request is wrong; retrying it is a loop.

The data model

CREATE TABLE idempotency_key (
    key           text        NOT NULL,
    user_id       bigint      NOT NULL,
    request_hash  text        NOT NULL,
    status        text        NOT NULL
        CHECK (status IN ('in_progress','completed','failed')),
    response_code int,
    response_body jsonb,
    locked_at     timestamptz NOT NULL DEFAULT now(),
    expires_at    timestamptz NOT NULL DEFAULT now() + interval '24 hours',
    PRIMARY KEY (user_id, key)
);

CREATE INDEX ON idempotency_key (expires_at);

The primary key is (user_id, key), not key alone. Two customers picking the same key must not collide, and — more importantly — a key must never let one user read another’s stored response. With UUIDs a collision is vanishingly unlikely; with a client that uses short keys it is not, and this costs nothing.

The race the obvious code loses

Here is the implementation almost everyone writes first:

async with db.transaction():
    row = await db.fetchrow(
        "SELECT * FROM idempotency_key WHERE user_id = $1 AND key = $2 FOR UPDATE",
        user.id, key,
    )
    if row:
        ...                                    # replay or 409
    await db.execute(
        "INSERT INTO idempotency_key (user_id, key, request_hash, status)"
        " VALUES ($1, $2, $3, 'in_progress')",
        user.id, key, request_hash,
    )

The FOR UPDATE looks like it serialises concurrent retries. It does not, because there is no row to lock.

Why the lock is not there

SELECT ... FOR UPDATE locks the rows it returns. On a first request it returns zero rows, so it takes no lock at all — there is nothing in the table to lock, and Postgres does not lock the absence of a row at Read Committed.

Two simultaneous first requests therefore both fall through:

T1  SELECT ... FOR UPDATE  -> 0 rows, no lock
T2  SELECT ... FOR UPDATE  -> 0 rows, no lock
T1  INSERT ... 'in_progress'
T2  INSERT ...             -> blocks on the primary key
T1  COMMIT
T2                         -> unique_violation -> 500

Run for real with asyncpg against PostgreSQL 18.4, two requests with one key via asyncio.gather and a 0.5 s provider call:

SELECT ... FOR UPDATE:   (201, 'created') and (500, 'UniqueViolationError'); 1 charge

The primary key does save you from charging twice, which is why this bug hides: the money is correct. What the client gets is a 500 on a request that may or may not have succeeded, which is precisely the situation idempotency existed to remove.

Insert first, then read

The fix is to stop asking whether the row exists and instead let the insert answer that question atomically:

row = await db.fetchrow(
    """
    INSERT INTO idempotency_key (user_id, key, request_hash, status)
    VALUES ($1, $2, $3, 'in_progress')
    ON CONFLICT (user_id, key) DO NOTHING
    RETURNING key
    """,
    user.id, key, request_hash,
)
if row is None:
    # Someone else owns this key. Now the row exists, so FOR UPDATE has
    # something to lock and we can safely inspect its state.
    return await replay_or_conflict(user.id, key, request_hash)

ON CONFLICT DO NOTHING ... RETURNING returns a row only if this statement created it. That single round trip decides ownership with no race: exactly one concurrent caller gets a row back and proceeds to do the work, and every other caller falls into the replay path.

This is the same shape as a compare-and-swap, and it generalises — whenever you are tempted to write “check whether it exists, then create it”, the check and the create need to be one statement.

The claim must commit before the side effect

Whether that insert works depends on which transaction it is in. The same two concurrent requests, then a crash between the provider call and the commit:

With the claim inside the request’s transaction (an ORM session or a FastAPI dependency that commits at the end), the uncommitted row makes every concurrent retry wait on the unique index, each holding a pooled connection for the whole provider call. Worse, a crash after the provider accepted the charge rolls the claim back with everything else, and the retry finds no key and charges again.

So the request runs in atomic phases (the design in Brandur Leach’s “Implementing Stripe-like Idempotency Keys in Postgres”): commit the claim; make the external call, passing a provider idempotency key derived from yours (f"{key}:charge"); commit the result.

A crash between phases leaves the row in_progress, which the stale-lock rule below recovers, and the provider’s own key stops the second charge. Multi-step handlers add a recovery_point column so a retry resumes after the last committed phase.

The three states, and the crash

in_progress ──success──> completed  (replay stored)
     │
     └────── failure ──> failed     (retry may try)

completed replays the stored response. in_progress means someone else is working right now — return 409, and let the client back off and retry, at which point it will find completed.

failed needs care. Stripe stores and replays the first result, including a 500, and saves nothing only when validation fails before any work or the request collides with a concurrent one. The rule behind it: a failure that happened before any side effect started may be retried with the same key; one that happened after a side effect may have taken place must be replayed (or reconciled), never re-executed.

The lock that outlives its owner

The state that strands the key is in_progress when nobody is working. The process handling the original request was deployed over, OOM-killed, or lost its database connection between the insert and the update. The row says in_progress forever, and every retry gets 409 forever. From the client’s side the operation is permanently wedged.

So in_progress must be time-bounded, not trusted:

STALE_AFTER = timedelta(seconds=90)

if row["status"] == "in_progress":
    if utcnow() - row["locked_at"] < STALE_AFTER:
        raise HTTPException(409, "request in progress")
    # Claim the stale lock: only one caller can win this update.
    claimed = await db.fetchrow(
        """
        UPDATE idempotency_key SET locked_at = now()
        WHERE user_id = $1 AND key = $2 AND status = 'in_progress'
          AND locked_at = $3
        RETURNING key
        """,
        user.id, key, row["locked_at"],
    )
    if claimed is None:
        raise HTTPException(409, "request in progress")

Note the AND locked_at = $3: the update only succeeds if nobody else re-claimed it since we read it. Without that comparison, several retries that all see the same stale lock will all claim it and all do the work.

Choose the timeout above your p99 handler duration, with margin. Set it too low and you re-execute work that was merely slow — which is the one outcome this whole mechanism exists to prevent.

Same key, different payload

if row["request_hash"] != request_hash:
    raise HTTPException(422, "Idempotency-Key reused with a different payload")

Do not silently replay the stored response. Consider a client that sends {"amount": 100}, then reuses the key with {"amount": 100000}. Replaying returns the successful 100 response, so the client believes the 100000 charge succeeded. It did not, and nobody finds out until reconciliation.

The hash must cover the body, path parameters and meaningful query parameters — and must exclude anything volatile: Date headers, trace ids, client timestamps. Hash a normalised, key-sorted JSON rendering, or two semantically identical requests will disagree.

Retention

DELETE FROM idempotency_key WHERE expires_at < now();

Twenty-four hours is the usual window — long enough that every legitimate retry has happened, short enough to bound the table. After expiry the same key is a new request: Stripe prunes keys once they are at least 24 hours old and treats a reused key after that as new.

Two rules follow: keep local keys at least as long as the client’s retry horizon, and reconcile stuck in_progress rows well inside the provider’s window, by looking the object up (by order id in its metadata), not by retrying with a key it no longer remembers.

Without a sweeper the table grows forever, and since every write path reads it, its degradation is felt everywhere.

Composing to end-to-end exactly-once

Idempotency keys make the request run once. They say nothing about the events that request emits. The full chain:

Client  POST /charges, Idempotency-Key: K
  API   K unseen -> do the work, and in the SAME
        transaction write the outbox row
  Relay reads outbox, publishes (at-least-once)
  Cons  dedupes on event id

Each layer solves its own redelivery problem, and together they give exactly-once effects over at-least-once delivery infrastructure. The critical line is “same transaction”: if the charge commits and the outbox write does not, the event is lost and no retry will produce it. See The transactional outbox.

Delivery semantics, hop by hop

At-most-once loses on a crash and never duplicates; at-least-once duplicates and never loses. Exactly-once exists only inside one closed system, a single database transaction or a Kafka transaction; everywhere else “exactly once” means at-least-once plus idempotence (Kafka delivery semantics).

Celery’s default, acknowledging just before execution, is at-most-once (What is Celery).

Where it breaks in production

A key minted per attempt

A retry wrapper (tenacity around the call that creates the key) or an SDK generates a fresh key on each attempt, so every retry is a new request and the whole mechanism does nothing. The key must be created once, stored with the operation, and reused.

Retries nobody sees

A proxy or service mesh configured to retry on connection errors re-sends a POST the client never retried. With a key, that is harmless; without one, it is a duplicate charge from infrastructure configuration.

The claim inside the request transaction

The measured case above: concurrent retries block holding connections, and a crash after the provider call double-charges. It hides because most requests never crash between the call and the commit.

What to watch

Replay rate, 409 rate, and the count of in_progress rows older than STALE_AFTER; the last one is the number of operations currently stuck.

The trade not taken

The alternative is to need none of this — design operations to be naturally idempotent, so retrying is harmless without bookkeeping. PUT /users/42 with a full representation is naturally idempotent. So is any operation keyed by something the client already owns: INSERT ... ON CONFLICT DO NOTHING on a client-supplied id needs no idempotency table at all.

Prefer that whenever the domain allows it. It is less machinery, less storage and fewer states. The idempotency-key table earns its complexity only when the operation has an external side effect you cannot re-run and cannot key — charging a card, sending an email, calling a partner API.

The other alternative, Redis instead of Postgres, is a real option for very high volume, and it trades correctness for throughput. A Redis SET NX is fast but is not in the same transaction as your business write, so a crash between them leaves the two disagreeing. Use Redis as a fast negative cache in front of the table, not as the record.

When at-most-once is the right contract

Sometimes a duplicate is worse than a loss: a marketing push, a “you were charged” SMS, a notification the next one supersedes; telemetry and heartbeats where the next message replaces this one. There, record the send before calling the provider: a crash loses the message rather than repeating it.

A one-time code is the opposite case: losing it blocks the user, so it gets at-least-once and a short validity window instead. Core NATS, Redis pub/sub and Synchronous Express Step Functions are at-most-once by design.

Related

• The transactional outbox - reliable event emission on the other side
• arq and dramatiq: the Celery alternatives - the same problem for queue consumers
• Idempotency - which HTTP verbs are idempotent by definition
• PUT vs PATCH: which writes are safe to retry - PUT and PATCH retry semantics