Event-driven agents
Most agent tutorials start from a chat box. Most production agents are not started by a person at all — a document lands, a ticket changes state, a payment fails, and the agent runs. That inversion changes the design.
event ──▶ consumer ──▶ agent run ──▶ result event
│
└──▶ tool callsThe agent becomes one consumer among many rather than the entry point, which is exactly the framing that separates “I built an agent” from “I ran agents in production”.
What changes when nobody is waiting
| Request-driven | Event-driven | |
|---|---|---|
| Latency budget | seconds, felt | minutes, tolerable |
| Failure is seen by | the user | nobody, unless you look |
| Retry | user presses again | the broker, automatically |
| Ordering | one at a time | whatever arrives |
The dangerous row is the second. A request-driven agent that fails produces a complaint. An event-driven one fails silently, and you find out from the business a week later. Alerting on the consumer, not just the agent, is the non-obvious requirement.
The third row is the one that causes damage: brokers redeliver, so your agent will run twice on the same event. That is not an edge case, it is the delivery guarantee.
At-least-once means idempotent runs
async def handle(event: Event) -> None:
# The event id is the natural idempotency key.
if await runs.exists(event.id):
return # already handled
await runs.start(
run_id=event.id, payload=event.data
)Dedupe on the event id, not on the payload — the same document can legitimately arrive twice with different ids, and two identical payloads with one id are a redelivery.
Beyond the run itself, the tools need their own idempotency, because a redelivery that gets past your check mid-run still repeats side effects. See Idempotency Keys (API-Level Deep Dive).
Picking the transport
| Broker | Fits when |
|---|---|
| SQS / Redis / RabbitMQ | task queue, no replay needed |
| Kafka | replay, ordering per key, many consumers |
| NATS JetStream | low latency, light ops, at-least-once |
The question that decides it is replay. If reprocessing a week of events through a new agent version is something you will want — and for anything where the agent’s output is a derived artefact, it is — you want a log, not a queue. See Kafka vs RabbitMQ — choosing a message broker.
Kafka’s per-key ordering matters here: partition by the entity the agent acts on (customer, document, account) so two runs never touch the same entity concurrently. That is cheaper than distributed locking and it is a design choice you make once.
Poison messages
An event that reliably crashes the agent will be redelivered forever, blocking the partition behind it. A dead-letter queue after N attempts is mandatory, not a refinement, and something has to actually read it.
For agents there is a second flavour: the event does not crash, but the agent loops to its step limit every time. Those are not errors to the broker, they are expensive successes. Alert on step-limit-reached as a rate.
Emitting events back
An agent that only calls tools is a leaf. An agent that publishes what it did becomes composable:
document.received
└─▶ [extract] ──▶ document.extracted
└─▶ [validate] ──▶ document.validated
└─▶ review.requestedEach stage is separately deployable, retryable and observable, and a human review step is just another consumer. This is the shape that scales organisationally as much as technically.
Gotcha: publish the outcome from the same transaction that records the run, via an outbox. Publishing directly from the agent means a crash after the publish and before the commit leaves downstream stages acting on a run your database has no record of.
Tracing across the hops
A trace that stops at the consumer boundary is nearly useless. Propagate the trace context in the event headers so one document’s journey through four agents is a single trace.
headers = {"traceparent": current_span().to_w3c()}This is ordinary W3C trace context, not an AI-specific mechanism — which is the point, and worth saying. See OpenTelemetry.
Related
Interview angle 5
- “How would you trigger an agent from a system event?” - the agent becomes a consumer rather than an endpoint: event in, run, result event out. The framing that matters is that it is one component among many, with a broker in front handling retry and ordering.
- “What breaks that a chat-triggered agent doesn’t?” - failures are silent because nobody is waiting, so you alert on the consumer as well as the agent. And delivery is at-least-once, so the same event will start the same run twice unless you dedupe on the event id.
- “Queue or log?” - decided by replay. If you will ever want to reprocess history through a new agent version, you need a log like Kafka; a queue throws the event away once acked. Partition by the entity the agent acts on so two runs never touch it concurrently.
- “What’s a poison message for an agent?” - two kinds. One crashes the run and is redelivered forever, which a DLQ after N attempts handles. The other completes but hits the step limit every time — not an error to the broker, just expensive. Alert on step-limit rate.
- “How do you keep a trace across four chained agents?” - propagate W3C trace context in the event headers. It is ordinary distributed tracing, not an AI mechanism, and without it each hop is a separate unrelated trace.