Durable Execution—Pause, Resume, Retry, Approve, and Compensate

Mission

By the end, you can design a long-running agent workflow that survives process loss; classify retryable failures; bind approvals to exact actions; and use idempotency and compensation to control uncertain side effects.

Prerequisite: Chapter 5.1 and the state, tool, approval, budget, and trace lab modules. Build artifact: a durable transition table and crash-window analysis for a simulated refund. Time: 75-95 minutes.

Before you read: Predict → Commit → Connect

An approver authorizes a $40 refund. The payment service applies it, but the worker crashes before recording success. On restart, should the agent call the service again?

Write yes, no, or “it depends.” Name the evidence needed to decide. Then list every point at which a crash could occur from proposal through final response.

A durable workflow is more than a long retry loop

A normal function keeps progress in process memory. If the process disappears, so does its stack. A durable workflow externalizes enough authoritative state to reconstruct the next safe transition.

Durability requires a contract for:

• identity: stable run, step, attempt, and operation IDs;
• state: persisted status, inputs, outputs, budgets, pending approvals, and receipts;
• transitions: allowed moves with compare-and-set or equivalent concurrency control;
• timers: deadlines and wake-up times that survive worker restarts;
• effects: idempotency, reconciliation, and compensation rules;
• versions: the code, prompt, schema, policy, and model under which a decision was made;
• evidence: events sufficient to explain what occurred without hidden chain-of-thought.

“Exactly once” is rarely a property of an end-to-end distributed action. A worker can lose a response after a remote service commits. Build around ambiguity with idempotency keys, provider lookups, durable receipts, and reconciliation.

Five different operations

Pause

Persist a waiting state and release the worker. Store why the run paused, what can resume it, the expiry, and the exact pending proposal. Do not keep a thread or browser session open for hours.

Resume

Receive a verified signal, reload state, check that the expected version still applies, and atomically claim the next transition. A stale approval or duplicate signal must not execute work twice.

Retry

Repeat a failed attempt only when the error is plausibly transient and the operation is safe to repeat. Bound attempts, elapsed time, tool calls, and cost. Use capped exponential backoff with jitter for shared dependencies, and stop when retries amplify overload rather than improve success.

Approve

Record a decision about a specific proposal: tool, normalized arguments, actor, scope, amount or resource, expiry, and policy version. Approval is not a reusable “yes” string. If arguments change, request a new decision.

Compensate

Perform a new business action that semantically offsets an earlier completed action: release a reservation, issue a refund, or restore an entitlement. Compensation is not time travel. It can fail, may not perfectly undo external consequences, and needs its own authorization, idempotency, retry, and evidence.

The names can differ, but ambiguous outcomes need a first-class state. Treating “timeout” as “nothing happened” is a duplicate-effect bug waiting to happen.

The crash-window analysis

For a side effect, examine at least four windows:

1. Before request: no remote effect; a claimed step may be safely released or retried.
2. Request in flight: the local system does not know whether the remote service received it.
3. Remote commit, response lost: the effect happened, but the caller saw a timeout.
4. Receipt stored, final checkpoint missing: the system can recover from the receipt without repeating the effect.

The safe protocol is application-specific, but a strong default is:

1. derive a stable operation ID before the call;
2. persist the authorized intent and operation ID;
3. call an API that honors that ID as an idempotency key when available;
4. persist the provider receipt and normalized result;
5. atomically advance the workflow state;
6. if the outcome is unknown, query by operation ID or send to reconciliation—do not guess.

An idempotency key helps only if the receiving service gives it defined semantics and retains deduplication records long enough. A random key regenerated on every attempt defeats the mechanism.

Retry classification

Classify errors at the adapter boundary, not from a model’s prose.

Retries multiply. If three nested layers each retry three times, one user request can produce up to 27 downstream attempts. Choose one layer to own retries for a dependency and expose attempt counts in traces.

Determinism, replay, and versioning

Some durable runtimes reconstruct state by replaying an event history through workflow code. In such systems, workflow decisions must be deterministic relative to recorded events. Network calls, model sampling, wall-clock reads, and random values belong in recorded activities or side-effect APIs, not unrecorded workflow logic.

Other systems persist explicit state-machine rows. They still need deterministic transition rules, optimistic concurrency, and version migration. A run waiting for approval during a deployment may resume under changed policy. Define whether it is pinned to the old version, revalidated under the new one, or cancelled. For risky actions, revalidation is usually safer.

Do not persist an opaque serialized framework object as the only record. Store portable business state and events so operators can inspect and migrate them.

Compensation is an ordered business plan

Suppose a trip workflow reserves a hotel, reserves a flight, then fails to arrange ground transport. Possible compensations are release flight and release hotel, often in reverse order. But each reservation may have different cancellation rules, deadlines, and fees.

Register the compensation intent before or atomically with the forward action where possible. Otherwise a crash after the forward action but before registering its compensation leaves an orphaned effect.

For every step, define:

• what counts as committed;
• the compensating action and its authority;
• whether compensation is mandatory, best effort, or impossible;
• how partial compensation is communicated;
• which human queue owns unresolved cases.

I do: make the simulated refund’s boundary explicit

Run the support capstone with and without approval:

cd courses/ai-agents/labs
python3 -m agent_lab.capstones.support T-1001
python3 -m agent_lab.capstones.support T-1001 --approve-refund

Both executions are intentionally offline. The first denies the proposal. The second records an approved simulation; the tool result still states executed: false. That distinction is the safety boundary.

For a real adapter, I would add a durable operation_id, an authorized-intent record, a provider idempotency key, an outcome-unknown state, and a reconciliation lookup. I would not change executed to true merely because a model or command-line flag requested it.

We do: recover a three-step booking

Together, fill this table for hold_flight → hold_hotel → confirm_itinerary:

Now introduce a crash after each request and after each receipt. If any row says “retry and hope,” it is unfinished.

You do: LAB-14 — Failure injection and transition design

Run the focused offline tests:

cd courses/ai-agents/labs
python3 -m unittest \
  tests.test_agent.AgentLoopTests.test_transient_tool_failure_retries_within_budget \
  tests.test_agent.AgentLoopTests.test_tool_attempt_budget_stops_retry_safely \
  tests.test_agent.AgentLoopTests.test_approval_policy_error_fails_closed \
  tests.test_capstones.SupportCapstoneTests.test_explicit_approval_only_records_simulated_proposal -v

Evidence boundary: runnable now versus design-only

• Runnable offline evidence: bounded transient retries, retry exhaustion, fail-closed approval-policy errors, default denial, and a non-executing simulated proposal.
• Design-only with the provided package: process-loss persistence, durable pause/resume, duplicate-signal arbitration, external idempotency and receipt lookup, crash recovery, and compensation. The package has no durable store or external transaction adapter.

Unless you add those components, treat the crash cases below as a reviewed transition-table exercise. Do not report them as executed fault-injection or recovery tests.

Then produce a transition table with columns:

current_state, event, guard, next_state, durable_write, external_effect, idempotency_key, receipt, retry_owner, compensation, operator_action.

Add cases for duplicate approval, expired approval, crash before call, timeout after call, duplicate resume signal, changed policy, exhausted budget, and failed compensation.

Done when with the provided lab: the runnable tests pass, every modeled crash window has a deterministic safe state, and the table labels durable execution, exact one-use approval, reconciliation, and compensation as unimplemented. Claim executable crash recovery only after an added implementation survives those fault-injection tests.

Pause & Recall

1. What must survive for a workflow to be durable?
2. Why is timeout after a write an ambiguous outcome?
3. What makes an approval replay-safe?
4. Why can compensation fail even when designed correctly?
5. How can nested retries amplify load?

Production lens

Monitor run age, time in state, retry attempts, reconciliation backlog, approval age, compensation failures, and stuck transitions. Alert on business impact, not merely queue depth. Provide operators with pause, cancel, retry-from-safe-state, reconcile, and compensate controls that themselves require authorization and create audit events.

Encrypt state and receipts, minimize sensitive payloads, and separate tenant data. Authenticate signals and callbacks. Prevent two workers from claiming the same transition with atomic leases or version checks. Preserve budgets across retry and resume; otherwise a crash becomes a way to obtain unlimited model calls.

Test upgrades against histories or state snapshots from earlier versions. Keep a rollback plan for workflow code and a forward-migration plan for runs already in progress. Backups are not enough: rehearse recovery while external effects continue to exist.

Chapter compression

• Durable workflows externalize authoritative state, transitions, timers, versions, and evidence.
• Pause releases compute; resume verifies a signal and atomically claims a transition.
• Retry only classified transient failures within one bounded owner.
• Bind approval to exact normalized arguments, identity, scope, expiry, and policy version.
• Ambiguous writes require idempotency and reconciliation, not assumptions.
• Compensation is a new fallible business action, not rollback magic.

Memory hook: Persist intent; identify effects; reconcile doubt; compensate explicitly.

Retrieval deck

• Q: What state follows a write timeout when the remote outcome is unknown?
  A: An explicit reconciliation or manual-review state—not automatic success, failure, or blind retry.
• Q: What must an approval identify?
  A: The exact action and normalized arguments, actor, scope, expiry, operation ID, and relevant policy version.
• Q: When is a retry safe?
  A: The failure is classified as transient, the operation is idempotent or known not to have happened, and attempts remain within budget.
• Q: Is compensation equivalent to database rollback?
  A: No. It is a separate business operation that may be partial, costly, delayed, or unsuccessful.

Spaced review

• Now: mark the four crash windows around one write.
• +1 day: reconstruct the pause/resume/retry/approve/compensate definitions.
• +3 days: design an idempotency and reconciliation contract for a payment API.
• +7 days: inject a duplicate signal into a state-machine design and prove one transition wins.
• +14 days: rehearse a compensation failure and produce the operator handoff evidence.

Sources and further study

• Temporal documentation: Durable execution concepts
• Garcia-Molina and Salem: Sagas
• Amazon Builders’ Library: Timeouts, Retries, and Backoff with Jitter
• Amazon Builders’ Library: Making Retries Safe with Idempotent APIs