Practical Case: Webhook Ingestion & Processing Service

Scenario

A third party (Stripe, GitHub, a partner API) sends you webhooks. You must receive them, verify they’re genuine, process them reliably, and never lose or double-apply one. The provider retries on any non-2xx, so your endpoint has to be fast and idempotent.

This is a classic senior take-home. The naive version — “parse the JSON and do the work in the handler” — fails on every axis that matters: it’s slow (provider times out and retries), it loses events (worker crashes mid-processing), and it double-applies (retry of an event you already handled). The interesting design is the receive/process split.

Stack: FastAPI + Postgres + Celery + Redis on AWS.

How would you design this?

Split it into two phases with a durable boundary between them.

1. Receive (synchronous, <50ms) — verify the signature, persist the raw event to Postgres, return 200 immediately. Do no business logic here.
2. Process (asynchronous) — a Celery worker picks up the event, runs the business logic, marks it done. Retries and failures happen here, invisible to the provider.

[provider] --POST--> [FastAPI: verify sig, INSERT raw event, 200]
                                   |
                                   v  (enqueue event id)
                              [Redis broker]
                                   |
                                   v
                          [Celery worker: process, mark done]
                                   |
                          [Postgres business tables]

The durable boundary is the webhook_events table. Once the row is committed, the event is safe — even if every worker is down, the event is not lost; it’s processed when workers come back.

The receive endpoint

@app.post("/webhooks/stripe")
async def receive(request: Request):
    body = await request.body()
    sig = request.headers.get("Stripe-Signature", "")

    if not verify_signature(body, sig, WEBHOOK_SECRET):
        raise HTTPException(400, "bad signature")

    event = json.loads(body)
    try:
        await db.execute(
            insert(webhook_events).values(
                # provider's event id = idempotency key
                id=event["id"],
                type=event["type"],
                payload=body,
                status="pending",
            )
        )
    except UniqueViolation:
        # already received — provider retry, ignore
        return {"status": "duplicate"}

    celery_app.send_task("process_webhook", args=[event["id"]])
    return {"status": "accepted"}

Key points:

• Verify the signature on the raw bytes, before parsing. Re-serialized JSON won’t match the HMAC. This is the #1 webhook bug.
• The provider’s event id is the idempotency key. A unique constraint on id makes duplicate receives a no-op — the provider retrying a webhook you already stored just hits the constraint.
• Persist before enqueue. If you enqueue first and the DB write fails, you have a task referencing a row that doesn’t exist. DB-write-then-enqueue means the worst case is a committed row that was never enqueued — recoverable by a sweeper (below).
• Return 2xx fast. Any slow work here = provider timeout = provider retry = load amplification.

The processing worker

@celery_app.task(
    name="process_webhook",
    bind=True,
    max_retries=5,
    # exponential: 1s, 2s, 4s, 8s, 16s
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True,
)
def process_webhook(self, event_id: str):
    with db.begin():           # one transaction
        event = db.query(WebhookEvent).filter_by(id=event_id) \
                  .with_for_update().one()

        if event.status == "done":
            # already processed — idempotent
            return

        handler = HANDLERS[event.type]
        # the business logic
        handler(json.loads(event.payload))

        event.status = "done"
        event.processed_at = func.now()

Key points:

• SELECT ... FOR UPDATE locks the row so two workers can’t process the same event concurrently (e.g. a duplicate enqueue, or a retry overlapping the original).
• Check status == "done" inside the lock. This is processing idempotency — distinct from the receive idempotency the unique constraint gave you.
• Business logic + status update in one transaction. If the handler half-succeeds and the process crashes, the transaction rolls back and the event stays pending — it gets retried cleanly. Never commit the business change and the status flip separately.
• At-least-once, made safe by idempotency. Celery (like SQS) is at-least-once. You don’t fight that; you make the handler safe to run twice. If the handler itself isn’t naturally idempotent (e.g. “send an email”), give it an idempotency key too.
• max_retries then dead-letter. After N failed retries, the task lands in a dead-letter state — set status="failed", alert, and move on. One poison event must not block the rest.

What about events that never get processed?

The persist-then-enqueue gap (DB committed, enqueue failed; or the broker dropped the message) leaves pending rows that no worker knows about. A sweeper closes the gap:

@celery_app.task
def sweep_stuck_events():
    stuck = db.query(WebhookEvent).filter(
        WebhookEvent.status == "pending",
        WebhookEvent.created_at < utcnow() - timedelta(minutes=5),
    ).limit(100)
    for event in stuck:
        celery_app.send_task("process_webhook", args=[event.id])

Run it every minute via Celery Beat. It re-enqueues anything stuck — and because processing is idempotent, re-enqueuing something that is actually in flight is harmless. This turns “exactly-once delivery” (impossible) into “at-least-once delivery + idempotent processing” (achievable).

On AWS — what runs where

A senior variant: drop Celery+Redis, use SQS directly. API Gateway → Lambda (verify + persist + SendMessage) → SQS → Lambda consumer, with an SQS DLQ. Fewer moving parts, native at-least-once + DLQ, no broker to operate. Celery wins if you need its scheduling/chaining or you’re already invested in it.

What can go wrong

• Signature verified on parsed JSON — re-serialization changes bytes, HMAC fails. Verify raw bytes.
• Business logic in the receive handler — slow response, provider retries, you’ve built a self-amplifying load problem.
• No idempotency — provider retries (and they will retry) double-apply. Unique constraint on receive, status check on process.
• Enqueue before persist — task references a missing row. Persist first.
• Committing business change and status flip separately — crash between them = double-apply or stuck. One transaction.
• Out-of-order delivery — webhooks are not ordered. A subscription.updated can arrive before subscription.created. Use the payload’s own timestamps/versioning, don’t assume arrival order.
• One poison event blocks the queue — bound retries, dead-letter, alert, keep moving.
• Replay attacks — an attacker resends a captured (validly-signed) webhook. Reject events whose timestamp is too old; the idempotency key also limits the blast radius.