Durable execution and human-in-the-loop

Article.

This is the strongest ground a backend engineer has in an AI interview, because none of it is really about models. An agent that pauses for a human approval tomorrow morning, survives a deploy, and does not charge the customer twice on retry is an ordinary distributed-systems problem wearing new vocabulary.

The problem

A naive agent is a function call. It holds state in local variables and dies with its process. That breaks the moment anything real happens:

• A deploy mid-run loses the conversation.
• A worker crash loses an hour of work and every tool call in it.
• An approval takes until tomorrow, and you cannot hold a process open.
• Someone asks what the agent did last Tuesday and there is no record.

An agent has to be a durable process, not a function call.

The mechanism: two ways to be durable

Every durable execution system is one of these, and the difference determines what constraints your code is under.

Snapshot

Serialise the state after each step. To resume, load it and continue.

step 1 -> [save state] -> step 2 -> [save state] -> crash
                                          |
                                    resume from here

Simple, and the constraint is obvious: everything in the state must be serialisable. No open connections, no file handles, no live clients. LangGraph’s Graph API snapshots like this after every superstep, but resuming re-runs the interrupted node from its first line, so within a node it behaves like replay; its Functional API (@entrypoint plus @task) re-executes the entrypoint and returns the stored results of tasks that already finished.

Replay

The workflow function is re-executed from the beginning, and calls that already completed return their recorded results instead of running again.

history:
  [started]
  [activity charge_card -> "ok"]
  [activity email       -> "sent"]

on resume the function runs from line 1:
  charge_card()  -> "ok" from history, no API call
  email()        -> "sent" from history, no send
  next_step()    -> no history; actually executes

This is Temporal’s model, and it buys enormous things — multi-week durability, first-class retries and timeouts — at the price of one hard constraint.

Why replay forces determinism

Replay only works if re-running the code takes the same path it took the first time. So workflow code must be deterministic:

# Breaks on replay. Each of these can diverge from the recorded history.
if random.random() < 0.1:          # different branch second time
    escalate()

deadline = datetime.now() + timedelta(hours=1)   # different value
rows = requests.get(URL).json()                  # unrecorded I/O

Take the first one. On the original run random() returned 0.05 and the workflow escalated, recording an escalate activity. On replay it returns 0.7, the code does not escalate, and the engine finds a recorded activity the code never asked for — a non-determinism error, and the workflow is stuck.

The engines give you deterministic substitutes:

now = workflow.now()                       # replayed from history
value = workflow.random().random()         # seeded from the run
data = await workflow.execute_activity(fetch, URL)   # recorded

Anything non-deterministic belongs in an activity, because activity results are recorded. That single rule is most of what “writing workflow code” means, and it is the thing interviewers probe when they ask about Temporal.

The crash window, and why idempotency is not optional

Both models share a gap that no amount of durability closes:

execute tool  ---->  [crash]  ---->  checkpoint written?  NO
     |
     side effect already happened

You executed the tool. You had not yet recorded that you executed it. On resume, the system does the only thing it can: run it again.

Durable execution gives you at-least-once, never exactly-once. The customer gets charged twice unless the charge itself is idempotent:

@activity.defn
async def charge_card(order_id: str, amount: int) -> str:
    info = activity.info()
    key = f"{info.workflow_id}:charge"           # same on every retry of this step
    return await payments.charge(order_id, amount, idempotency_key=key)

# in the workflow
await workflow.execute_activity(
    charge_card, args=[order_id, amount],
    start_to_close_timeout=timedelta(seconds=30),
)

Temporal retries the activity, so the key is derived inside it from the workflow id and the step, and passed to the payment provider, which is the system that actually deduplicates.

This is the same conclusion as every message-driven system, arrived at from a different direction — see Idempotency keys.

Checkpointing in practice

from langgraph.checkpoint.postgres import PostgresSaver

with PostgresSaver.from_conn_string(DSN) as checkpointer:   # a context manager
    checkpointer.setup()                       # once per deploy: creates tables
    graph = builder.compile(checkpointer=checkpointer)
    config = {"configurable": {"thread_id": "tenant-7:case-42"}}
    graph.invoke({"messages": [msg]}, config)   # starts, or resumes mid-flight

A checkpoint holds the message history, scratch state, which node comes next, and enough metadata to reconstruct the run.

The underrated benefit is not crash recovery — it is time-travel debugging. Rewind to any step, change the state, re-run from there. For a non-deterministic system that is a materially better story than reading logs.

Human-in-the-loop

The pattern that requires durability, because you cannot hold a process open while someone approves a refund tomorrow.

def approve(state):
    decision = interrupt({
        "action": "refund",
        "order_id": state["order_id"],
        "amount": state["amount"],
        "reason": state["reason"],          # why the agent chose this
    })
    if decision["approved"]:
        return {"result": execute_refund(state)}
    return {"result": f"Denied: {decision.get('note', 'no reason given')}"}

# Hours later, a different process, from a web handler:
graph.invoke(Command(resume={"approved": True}), config)

interrupt persists the state and returns control. Anything holding the thread_id can resume the run.

Where to put the gate

Approve-before-acting is the default to name. But reviewing the plan is the cheapest intervention — catching a wrong approach before twenty tool calls beats correcting the output after.

The approval payload is the whole design

Give the reviewer enough to decide without re-deriving the agent’s reasoning: the action and its arguments, why the agent chose it, what it will affect, and what happens if denied.

An approval UI showing {"tool": "refund_order", "args": {...}} and nothing else forces the reviewer to guess — and a reviewer who cannot evaluate the request approves everything, which is worse than no gate at all because it manufactures an audit trail of meaningless approvals.

Where it breaks in production

Version skew. A run checkpointed under graph v1 resumes under v2, and the node it wanted no longer exists. With replay-based engines it is sharper: the recorded history must still match what the new code decides, so any logic change to a workflow with runs in flight is a potential non-determinism error.

The engines provide a versioning primitive for exactly this:

if workflow.patched("add-fraud-check"):
    await execute_activity(fraud_check, order_id)   # new runs only

Old runs replay the un-patched path; new runs take the new one. The alternative policies are draining — stop starting runs, wait for in-flight ones to finish, then deploy — or migrating checkpoints, which is real work. Decide which before you need it, because the decision arrives during an incident otherwise.

Orphaned interrupts. A run pauses for approval and nobody ever responds. It sits in the store indefinitely, holding state, invisible. Every interrupt needs a timeout and an escalation path, and someone needs a dashboard of runs waiting on a human — otherwise the failure mode is silence.

PII in checkpoints. Checkpoints persist message history, which contains whatever the user typed. That is a personal-data store: it needs encryption at rest (LangGraph’s EncryptedSerializer, AES with a key from LANGGRAPH_AES_KEY, passed as serde= to a saver built on your own connection), a retention policy, and an answer for deletion requests (checkpointer.delete_thread(thread_id)).

Most teams discover this when an auditor asks.

Unbounded growth. Checkpoints accumulate per step per run, and the open-source Postgres saver has no TTL: retention is a scheduled job that calls delete_thread() for threads past their age (hosted LangSmith Deployment offers TTLs). Keep a compact audit summary for longer than you keep the full state.

LangGraph or a workflow engine

The defensible position is both, at different levels. LangGraph for the agent loop, where LLM ergonomics matter and the state is a message list. A workflow engine for the long-running business process that contains the agent, where retries, compensation and multi-week durability are the point.

For an agent orchestrating a refund across payment, inventory and notification services, that split is exactly right — and saying so shows you see the agent as one component in a system rather than as the system.

The trade not taken

All of this is machinery, and plenty of agents do not need it. If a run completes in thirty seconds, has no side effects, and a failure just means the user retries, then durable execution is infrastructure you are carrying for no benefit — a request handler and a timeout will do.

The threshold is concrete: durability earns its cost when a run outlives a process. Approvals, multi-minute runs, side effects that cannot be repeated, or an audit requirement. Below that line, the state belongs in the request.

Related

• The agent loop - the loop being made durable
• Sagas and compensation - compensation when step 7 of 10 fails
• Temporal vs Celery vs Other Orchestrators - the engine side