Long-running agent jobs

An agent run takes seconds to minutes, occasionally longer. That does not fit a request/response endpoint, and the shape you choose for it is a system design question an interviewer can go a long way into.

Three shapes, and when each is right

Synchronous is fine for a single LLM call and stops being fine the moment a tool loop is involved. A load balancer with a 30-second idle timeout will kill your agent mid-run and return a 502 that tells the user nothing.

Streaming solves the perception problem, not the duration problem — see The chat API surface. It also pins a connection open for the whole run, which caps your concurrency at however many sockets you can hold.

Job is the honest answer past a minute or so.

The job shape

@app.post("/runs", status_code=202)
async def start(req: RunRequest) -> RunAccepted:
    run_id = uuid7()
    await queue.enqueue(
        "agent.run", run_id=run_id, payload=req
    )
    return RunAccepted(
        run_id=run_id, status_url=f"/runs/{run_id}"
    )

@app.get("/runs/{run_id}")
async def status(run_id: UUID) -> RunStatus:
    return await store.get(run_id)

202 Accepted plus a status URL is the standard contract. The client polls, subscribes, or gets a webhook — and the API stays fast regardless of how long the agent thinks.

Telling the client it finished

Polling with sensible backoff is underrated and is usually where to start. Webhooks are the right answer between services, and they bring their own requirements — signature verification, retries, and an idempotent consumer, because you will deliver twice eventually.

Gotcha: a webhook that fires before the run’s state is committed gives the client a callback for a run your own API then reports as pending. Commit first, then publish — or use the outbox pattern. Same bug as any write-then-notify system.

Progress is not optional

A run with no visible progress is indistinguishable from a hung one, and the first support ticket will be “it’s stuck”.

Emit a step event per loop iteration — which tool, how long, what it returned in summary form. That gives you three things at once: a progress bar, a debug trail, and the spans your observability needs anyway. See LLM observability.

Resumption, and what “resume” actually means

If the worker dies at step 7, you want to continue rather than restart — partly for latency, mostly because the first six steps had side effects and cost money.

That requires the state to be outside the process: checkpointed after each step, keyed by run id. LangGraph’s checkpointers do this for the agent graph; a workflow engine does it for the whole process. The trade-off is covered in Durable execution and human-in-the-loop.

The part people miss: resuming replays tool calls unless they are idempotent. A crash between “charge the card” and “write the checkpoint” means the resumed run charges again. That is the same problem as any at-least-once system, and the same fix — idempotency keys, deduped by the tool, not by the agent.

Cancellation

Users close tabs. A run nobody is waiting for still burns tokens.

for step in range(MAX_STEPS):
    if await store.is_cancelled(run_id):
        await store.mark(run_id, "cancelled")
        return
    ...

Check between steps rather than trying to interrupt an in-flight provider call. Cancelling mid-call is possible but leaves you unsure whether the provider billed you, so a step boundary is the clean seam.

Pair it with a wall-clock deadline for the whole run, not just per call. An agent that takes twelve minutes has usually gone wrong, and the step limit alone will not catch a run where each step is slow rather than numerous.

What to store

A run record earns its keep when something goes wrong:

run_id, tenant_id, status
created_at, finished_at
input, output, error
steps[], model
tokens_in, tokens_out, cost
trace_id

trace_id is the one to insist on — it is what turns “this run was wrong” into an actual investigation rather than a re-run and a shrug.

Related

• Durable execution and human-in-the-loop — checkpointing and workflow engines
• Concurrency and backpressure — how many of these you run at once
• Agent failure modes — what goes wrong inside the loop
• Idempotency Keys (API-Level Deep Dive) — the mechanism resumption depends on