Webhooks

A webhook is an HTTP callback: instead of you polling a provider, the provider POSTs to a URL you own when something happens. Cheap to describe, and the source of a specific set of production failures that interviewers probe for because almost everyone has been burned by at least one.

Two sides, and they have different problems. Receiving is about trust, duplicates and speed. Sending is about retries, ordering and consumers who are down.

Receiving: the four rules

Verify the signature first

1. Verify the signature before you parse the body. The URL is public, so anyone can POST to it. Providers sign the raw body with a shared secret; you recompute and compare.

import hashlib
import hmac
import time

TOLERANCE = 300  # seconds

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    ts, sig = parse_header(header)          # "t=1699…,v1=abc…"
    if abs(time.time() - int(ts)) > TOLERANCE:
        return False                        # too old: a replay
    expected = hmac.new(
        secret.encode(), f"{ts}.".encode() + raw_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, sig)

Three details that are each a real bug:

• Sign the raw bytes, not the parsed object. json.dumps(json.loads(body)) reorders keys and changes whitespace, so the signature will not match.
• hmac.compare_digest, never ==. String comparison short-circuits on the first differing byte, which leaks the secret to a timing attack.
• Include a timestamp and reject old ones. Without it, a captured request can be replayed forever — the signature stays valid.

Assume duplicates

2. Assume duplicates. Delivery is at-least-once everywhere. A provider that does not get a 2xx in time retries, including when your handler succeeded and the response was lost. Deduplicate on the provider’s event id.

async def handle(event_id: str, payload: dict, conn) -> None:
    # The unique index is what makes this safe under concurrency,
    # not the SELECT - two workers can pass a check at the same time.
    inserted = await conn.fetchval(
        "INSERT INTO webhook_events (id, received_at) VALUES ($1, now()) "
        "ON CONFLICT (id) DO NOTHING RETURNING id",
        event_id,
    )
    if inserted is None:
        return                              # already processed
    await enqueue(payload)

Return fast, and distrust the payload

3. Return 200 fast, work later. Providers time out in seconds and treat slow responses as failures. Validate, persist, enqueue, respond — do the real work in a background consumer. A handler that calls three APIs inline will be retried under load, multiplying the load.

4. Do not trust the payload’s contents. Webhook bodies are attacker- influenced input even when signed, because the signature proves who sent it, not that the data is true. For anything that matters — a payment amount, a subscription tier — treat the webhook as a notification that something changed and re-fetch the object from the provider’s API.

Sending: what your consumers need

If you emit webhooks, you own the retry policy and the contract.

Retries need a budget and an end. Retry forever and a permanently-broken consumer becomes an unbounded queue. The usual shape is roughly a day of exponentially spaced attempts, then park the event and notify the customer.

Ordering is the one people get wrong. Parallel delivery means event 2 can arrive before event 1, and a retry of an old event can arrive after a newer one. Do not promise ordering. Instead put a monotonic sequence or a timestamp in the payload so consumers can discard stale state, and design events to be order-independent where you can.

Give consumers a replay tool. An endpoint or dashboard button that redelivers the last N events converts an outage into a self-service recovery, and it is the feature that support teams ask for first.

Webhooks vs polling

The honest trade is that webhooks fail silently and polling fails loudly. If your endpoint is down for an hour, the events are gone unless the provider retries long enough. Robust integrations use both: webhooks for latency, and a periodic reconciliation sweep that pulls anything the webhook path missed. Being able to propose that pairing unprompted is the senior answer.

Where it breaks in production

• Signature verified against a re-serialised body. Works in tests where the fixture round-trips cleanly; fails on the first payload with a unicode escape.
• No dedup, so a retry double-charges. The classic.
• Slow handler, so the provider retries mid-processing and two workers run the same event concurrently. The unique index saves you; a SELECT first does not.
• Secret in the URL as a query parameter, then logged by every proxy in the path. Use a header and a signature.
• No local development path. Webhooks cannot reach localhost, so people test in production. A tunnel (ngrok and similar) or the provider’s CLI replay is the fix, plus recorded fixtures in tests.
• Silent schema drift. A provider adds a field, or changes an enum, and your strict parser starts rejecting. Parse permissively, log unknown fields.

Related

• Idempotency - idempotency keys, the same problem from the caller’s side.
• Idempotency keys
  • the storage design.
• Timeouts, retries and backoff
  • backoff and jitter.
• Dead-Letter Queue (DLQ) for Celery
  • parking what will not deliver.
• The transactional outbox
  • emitting events without losing them.
• Secrets management and rate limiting
  • holding the signing secret.
• REST principles - REST constraints and resource modelling