Worked Design — Idempotent Payment Flow

Article.

“Design a payment processing flow.” The whole question is correctness under failure — never double-charge, never lose a payment, stay consistent across your DB and a third-party processor you don’t control. Stack: FastAPI + Postgres + Celery/SQS + Stripe-style processor.

1. Requirements

Functional: a client initiates a payment; the system charges via a payment processor (Stripe/Adyen); the result (success/failure) is recorded and reflected to the client and downstream (order fulfilment, receipts).

Non-functional — these are the design:

Never double-charge — a retry, a network blip, a double-click must not charge twice.
Never lose a payment — if money moved, the system must know.
Consistency across boundaries — your DB, the processor, and downstream services must converge to the same truth even when steps fail mid-flight.
Auditable — every state transition is recorded.

This is a correctness problem, not a scale problem. There’s no clever caching answer — the entire job is handling partial failure.

2. The core threats

text

client ──► your API ──► processor ──► money moves
              │            │
         crash here?   timeout here: did it charge?
              │            │
         client retry? DB write fails after the charge

Every arrow can fail, and a failure between “Stripe charged the card” and “your DB recorded it” is the dangerous one — money moved but your system doesn’t know.

3. Idempotency keys — the foundation

The client generates a unique idempotency key per payment intent (not per HTTP attempt) and sends it with every retry of that payment.

http

POST /payments
Idempotency-Key: 7f3a9c1e-...        ← same key on every retry of THIS payment
{ "amount": 5000, "currency": "usd", "order_id": "ord_42" }

The server side

sql

CREATE TABLE payments (
    merchant_id     BIGINT NOT NULL,
    idempotency_key TEXT   NOT NULL,
    request_hash    BYTEA  NOT NULL,      -- SHA-256 of the normalised request
    order_id        TEXT   NOT NULL,
    amount_minor    BIGINT NOT NULL,      -- cents; INT caps at ~$21.5M
    currency        CHAR(3) NOT NULL,     -- ISO 4217
    status          TEXT   NOT NULL,      -- see the state machine below
    claimed_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    processor_charge_id TEXT,             -- Stripe's id, once we have it
    response_body   JSONB,                -- the stored response to replay
    PRIMARY KEY (merchant_id, idempotency_key)   -- keys are scoped per client
);

On each request

INSERT ... ON CONFLICT (merchant_id, idempotency_key) DO NOTHING.
Conflict (key already exists) → this is a retry. If the stored request_hash differs, the client reused a key for a different payment: reject with 422 rather than replaying the old answer. If the row is succeeded/failed, return the stored response. If it is still pending, compare claimed_at with a lease (say 60 s): a fresh claim means the original is in flight (409); an expired one means the first attempt crashed, and exactly one caller takes it over with UPDATE ... SET claimed_at = now() WHERE ... AND claimed_at < now() - interval '60 s'.
No conflict (we inserted) → this is the first time; proceed to charge.

The unique constraint is the hard guarantee that two concurrent requests for the same payment cannot both proceed. Twenty threads sending the same key at once (SQLite, the same ON CONFLICT ... DO NOTHING):

text

20 concurrent requests, same key: 1 inserted the claim, 19 saw a conflict

No lock, no check-then-insert: the database’s uniqueness check is the serialization point, and the one insert that wins is the only caller allowed to charge. The full race, including why a SELECT first loses it, is in Idempotency keys.

And pass that same key through to the processor — Stripe et al. accept an Idempotency-Key header and dedup on their side too. Belt and suspenders: even if your dedup has a gap, the processor won’t double-charge.

4. The flow — and the partial-failure handling

text

1. INSERT row (pending), keyed by idempotency key
   -- the atomic claim; committed before step 2
2. charge the processor with the SAME key
3. success: UPDATE status=succeeded, charge id, response
   failure: UPDATE status=failed, response
4. return the result; emit payment.succeeded downstream

Now the failure cases:

Crash between step 1 and step 2 — row is pending, no charge happened. A retry (same key) sees pending and gets 409 until the row is stale; then one caller reclaims it (a guarded update on claimed_at), or a reconciliation job notices the stale pending and resolves it.
Processor call times out (step 2) — you don’t know if the charge succeeded. Do not blindly retry — that risks a double charge. The row stays pending. Resolution: retry the processor call with the same processor idempotency key while it is inside the processor’s key window (Stripe keeps keys at least 24 h and replays the saved result, even a saved error); after that window, find the charge by your own reference in its metadata (order_id). Stripe has no lookup by idempotency key. Reconcile the row from the answer.
Crash between step 2 and step 3 — money may have moved but the DB still says pending. The reconciliation job (below) catches this.
Step 3 DB write fails after a successful charge — same: money moved, DB out of sync. Reconciliation catches it. This is why the idempotency key is also sent to the processor and stored — it’s the join key for reconciliation.

The principle: a pending row is a question, not a final state. Something must always resolve it — a retry, or reconciliation.

The timeout, simulated

A processor stand-in with the two properties that matter (it dedups on an idempotency key and replays the saved result; charges are searchable by your own metadata), charging, then losing the response:

text

blind retry, new request: charges for ord_42 = ['ch_1', 'ch_2']
retry with the same key: charges for ord_42 = ['ch_1']

The blind retry charged the customer twice and nothing raised. The keyed retry got the first charge back. The search by order_id is what reconciliation uses once the processor’s key window has passed.

5. Reconciliation — the safety net

A scheduled job that closes the gap between your DB and the processor’s reality:

Find payments rows stuck in pending past a threshold (e.g. 5 minutes).
For each, ask the processor: replay the call with the same idempotency key inside its key window, or search by order_id in metadata after it.
Update the row to match reality (succeeded with the charge id, or failed).
Additionally: periodically pull the processor’s list of charges and assert every one has a matching row — catches “money moved, we have no record at all.”

Reconciliation is non-negotiable in payments. Idempotency keys prevent double-charges; reconciliation prevents lost payments. You need both.

6. Webhooks — the processor’s async truth

Many processor outcomes arrive asynchronously via webhook (a charge that was pending settles later; a dispute; an async payment method completing). The webhook handler must itself be idempotent:

Webhooks are delivered at-least-once — the processor retries until you 200. So the same charge.succeeded event can arrive twice.
Dedup on the processor’s event id (INSERT ... ON CONFLICT on event_id).
Verify the webhook signature and its timestamp — the endpoint is public; an attacker could POST a fake “payment succeeded”, or replay a real one captured earlier. Stripe signs t=<timestamp> into the signature, and its SDK rejects events older than a tolerance (300 s by default).
Process the webhook → update the payment row → emit the internal event. Treat the webhook as another path that resolves a pending row, alongside the synchronous response and reconciliation.

7. Downstream — don’t dual-write

After a payment succeeds you must tell other services (fulfil the order, send a receipt). Doing UPDATE payment then publish event as two separate steps is the dual-write problem — a crash between them loses the event.

Use the transactional outbox: in the same DB transaction that marks the payment succeeded, post its double-entry journal (The double-entry ledger) and insert an outbox row for the payment.succeeded event. A relay process publishes outbox rows to the queue. Downstream consumers are idempotent (keyed on payment id). Now the payment status and the event are atomic — both happen or neither.

8. State machine

Model the payment as an explicit state machine with the real processor states, not just three: pending → requires_action (3-D Secure / SCA, completed by the customer later) → authorized → captured, or → failed / canceled. Refunds (full or partial) and disputes are separate objects that point at the payment, each posting a reversing ledger entry rather than editing the payment. No transition goes backwards, and webhooks can arrive out of order, so on each event fetch the object’s current state from the processor instead of trusting event order. Every transition is a recorded, audited row change. This makes the system reason-about-able and is itself an interview talking point: “I’d model it as an explicit state machine so illegal transitions are impossible and every change is auditable.”

9. Bottlenecks & trade-offs

Correctness over latency — the synchronous flow is deliberately not the fastest path; the client waits for a real answer or a clear “in progress.” That’s correct for payments — never trade correctness for latency here.
The timeout ambiguity is unavoidable — a timed-out processor call is genuinely “unknown”; the system’s job is to resolve the unknown safely (query by key, reconcile), never to guess (blind retry).
Idempotency key TTL — keep your keys for days; they are cheap. But the processor forgets its copy sooner (Stripe can prune keys once they are 24 hours old), and a retry after that is a new request upstream. Retries older than the processor’s window must go through reconciliation (search by your order_id), never a re-send.
Scale — payments QPS is usually modest; this rarely needs sharding. If it did, shard payments by idempotency key — every operation for one payment is keyed by it.
Don’t roll your own processor integration loosely — lean on the processor’s idempotency keys and on searching charges by your own reference in metadata; there is no lookup by idempotency key.

Where it breaks in production

A key minted per attempt. A client that generates a new key on each retry has no idempotency at all; the key belongs to the payment intent, created once and stored client-side before the first attempt.
A re-send after the processor forgot the key. Your table keeps keys for days, the processor for about 24 hours. A retry after that is a brand-new charge upstream; past the window, resolve by searching on order_id, never by re-sending.
pending rows nobody resolves. Without the reconciliation job, every crash between the charge and the commit is money taken and an order never fulfilled. Alert on the count and age of pending rows.
Webhooks trusted in order. charge.refunded can arrive before charge.succeeded; fetch the current object instead of applying events in arrival order.
The claim inside the request transaction. If the pending insert is rolled back with a failed request, a retry finds no claim and charges again. Commit the claim before calling the processor.

The trade not taken: a lock around the charge

The tempting design wraps the processor call in a distributed lock keyed by order: acquire, charge, record, release. A lock with a TTL fails exactly when the charge is slow, which is the timeout case this whole design exists for: the lock expires during a hanging processor call, a retry acquires it and charges again. That failure is shown in Redlock and the Kleppmann Critique.

The design above needs no lock. The unique row decides who may charge, the processor’s idempotency key makes a repeat harmless, and reconciliation resolves whatever is left unknown. Each piece is a durable record, not a lease that can expire.

Interview angle 7

“How do you prevent double-charging?” — a client-generated idempotency key per payment intent, stored with a unique constraint: INSERT ... ON CONFLICT DO NOTHING. A retry hits the existing row — return the stored response, don’t charge again. Pass the same key to the processor so it dedups too.
“The call to Stripe times out — you don’t know if it charged. What do you do?” — do not blindly retry (double-charge risk). Leave the row pending and resolve the unknown: retry with the same processor idempotency key inside its window (the processor replays the saved result), or search for the charge by order_id metadata after it, and reconcile the row from the answer.
- Follow-up: show the difference. Simulated: the blind retry left two charges for the order, the keyed retry one, the first charge replayed.
- Follow-up: why not a lock around the charge? Its TTL expires during the slow processor call it was meant to protect, and the retry charges again. The unique row and the processor’s key need no lease.
“Two requests with the same key arrive at the same moment?” — the unique constraint serializes them: of 20 concurrent inserts, 1 claimed and 19 saw a conflict.
“How do you make sure a payment is never lost?” — a reconciliation job. Idempotency keys stop double-charges; reconciliation stops lost payments. It sweeps stale pending rows, resolves each by replaying with the same key inside the processor’s window or searching by order_id after it, and converges your DB to the processor’s reality. A pending row is always a question something must answer.
“How do you handle the payment webhook?” — verify the signature (public endpoint), dedup on the processor’s event id (webhooks are at-least-once), then treat it as another path that resolves the payment row — alongside the sync response and reconciliation.
“How do you tell downstream services without losing the event?” — transactional outbox: insert the payment.succeeded event into an outbox table in the same transaction as the status update. A relay publishes it; consumers are idempotent. Avoids the dual-write problem (crash between DB update and publish loses the event).
“Why model it as a state machine?” — explicit pending → succeeded/failed transitions make illegal states impossible, every change is an audited row, and the whole flow becomes reason-about-able. Payments demand auditability and correctness over cleverness.

Worked Design — Idempotent Payment Flow

Worked Design — Idempotent Payment Flow

1. Requirements

2. The core threats

3. Idempotency keys — the foundation

The server side

On each request

4. The flow — and the partial-failure handling

The timeout, simulated

5. Reconciliation — the safety net

6. Webhooks — the processor’s async truth

7. Downstream — don’t dual-write

8. State machine

9. Bottlenecks & trade-offs

Where it breaks in production

The trade not taken: a lock around the charge

Related

Interview angle 7