Production Operations—Deployment, Scaling, Latency, Cost, and Release Safety

Production is not the place where an agent finally becomes real. It is where every hidden assumption becomes traffic, delay, cost, and an incident someone must own.

Mission

By the end, you can turn a bounded agent into an operable service: define service objectives, separate request admission from execution, scale without duplicate actions, measure latency and cost per successful outcome, and release or roll back the entire agent harness safely.

Prerequisites: Chapters 1-18 and a passing offline lab suite. Build artifact: a production operations card, release gate, and rollback rehearsal for one capstone. Time: 90-120 minutes.

Before you read: Predict → Commit → Connect

Northstar's research agent works in staging. At launch, traffic rises tenfold. The team adds ten workers, yet users wait longer, provider errors rise, and a retried request publishes two reports.

Before reading, write three predictions:

1. Which resource became the real bottleneck?
2. Why did more workers make the system worse?
3. Which invariant should have prevented duplicate publication?

Commit to your answers. Production engineering improves when a hypothesis is written before the graph is inspected.

Operate a queueing system, not a demo loop

An interactive demo hides arrivals, concurrency, dependency quotas, cancellations, and partial failure. A production service needs two distinct planes:

• Admission plane: authenticates, authorizes, validates, assigns an idempotency key, applies per-tenant limits, estimates work, and either accepts, queues, or rejects it.
• Execution plane: leases accepted work, restores a checkpoint, calls models and tools within budgets, records observations, and commits effects through narrow gates.

Keep the user-facing request short when the task may be long. Return a run identifier and status contract, then execute asynchronously. A provider's background-processing feature can help with one model call, but it does not replace your run state, authorization, cancellation, or recovery design.

The queue is a control surface. It absorbs bursts, exposes backlog, enables fair scheduling, and lets workers scale independently. It is not infinite storage. Define maximum age, maximum depth, priority policy, tenant fairness, cancellation semantics, and what users see when capacity is exhausted.

Capacity begins with the constrained dependency

Let arrival rate be \(\lambda\) tasks per second and sustainable service rate be \(\mu\) per worker. The arithmetic \(workers \geq \lambda/\mu\) is only a starting estimate. Real agent runs have variable step counts, correlated bursts, and shared limits:

• model requests or tokens per interval;
• tool API concurrency and quotas;
• database connections and lock contention;
• queue partitions and worker memory;
• human approval capacity;
• per-tenant budgets.

When utilization approaches saturation, small variations can create large queues. Do not autoscale only on CPU if workers mostly wait on remote calls. Useful signals include oldest-job age, ready-queue depth, active leases, provider throttles, tool saturation, and completion rate. Scale gradually, bound concurrency at every dependency, and use backpressure before overload cascades.

Retries increase load precisely when a dependency is unhealthy. Retry only transient failures, with capped attempts, jittered backoff, and a total deadline. Respect server retry guidance. A retry must not repeat a non-idempotent effect. Assign the business operation an idempotency key and store its outcome atomically enough that the same request returns the first result.

Latency is a distribution and a path

An average hides the users waiting in the tail. Measure at least p50, p95, and p99 end-to-end latency, plus time in queue and each major span. Segment by task type, model, tool, region, tenant class, and outcome. Never compare a fast refusal with a successful ten-step research run as though they were the same workload.

Decompose the critical path:

\[ L_{total}=L_{queue}+\sum L_{sequential}+\max(L_{parallel})+L_{approval}+L_{commit} \]

The equation is a map, not a promise. Measure it from traces. Then choose the intervention that targets the dominant term:

• remove model calls that deterministic code can perform;
• select the smallest model that passes the task eval;
• reduce output tokens and irrelevant context;
• parallelize independent reads, never dependent writes;
• cache only when identity, tenant, freshness, and invalidation are explicit;
• stream safe progress or partial output when it improves the experience;
• move long work behind a status endpoint and cancellation control.

Optimize against quality and safety gates. A cheaper, faster answer that fails grounding is not an optimization.

Cost belongs to the outcome, not the call

Record cost at the run level with the price schedule and model version used at that time. Include model input, cached input, output and reasoning usage where exposed; tool fees; retrieval; compute; storage; human review; retries; and abandoned or failed runs.

Two metrics are especially useful:

\[ Cost\ per\ completed\ run=\frac{total\ operating\ cost}{completed\ runs} \]

\[ Cost\ per\ successful\ outcome=\frac{total\ operating\ cost}{runs\ that\ pass\ the\ task\ criterion} \]

The second prevents a false victory from a cheap system that completes the wrong task. Report distributions, not just totals. Set a maximum run budget, daily tenant budget, and anomaly alert. When a run crosses its budget, stop at a defined safe state; do not merely stop tracing the spend.

For the offline lab, provider cost is zero. Use model attempts, tool attempts, tool output characters, and approval-policy decisions as cost proxies. The included approvers are deterministic test policies, not human-review labor. Label these measures as proxies—never present them as currency.

Release the harness, not only the prompt

Agent behavior depends on a bundle: application code, model snapshot, system instructions, tool schemas, policy rules, retrieval corpus and index, memory migrations, budgets, eval set, and provider configuration. Give that bundle one immutable release identifier.

A safe progression is: offline eval → shadow traffic → small canary → staged expansion → full release. Shadow execution must disable external effects, but it can still process real user data; authorize, minimize, isolate, retain, and delete that data under the same privacy contract as production. At every stage compare candidate and control on task success, safety violations, tool/approval behavior, latency, cost, and trace completeness. Predetermine stop conditions. A human should not improvise whether a leak is “small enough” during rollout.

Rollback means more than redeploying old code. Decide what happens to in-flight runs, new-format checkpoints, already approved proposals, changed indexes, and partially committed effects. Prefer forward-compatible state, reversible migrations, feature flags for risky capabilities, and a kill switch that disables write tools independently of read-only service.

I do: an operations card for the research capstone

The current research lab is synchronous, deterministic, local, and read-only. Its contract is five steps, five tool calls, seven attempts, two bounded tools, and a cited answer or explicit no-match result.

I would productionize that contract as follows:

Notice what I did not do: invent a throughput target without workload measurements.

We do: build a canary gate

Run the current capstone test twice, once as the control and once after a harmless prompt edit:

cd courses/ai-agents/labs
python3 -m unittest tests.test_capstones.ResearchCapstoneTests -v

Together, define the candidate gate before looking at results:

1. both grounding cases pass;
2. tool calls remain within five;
3. required run and tool trace events remain present;
4. no new sensitive field appears in trace payloads;
5. the no-match response remains explicit;
6. the release identifier is attached to the report.

Then add two operational observations on paper: queue age and per-run cost proxy. The local runner has no durable queue or checkpoint, so mark those controls not implemented rather than awarding imaginary evidence.

You do: LAB-19 — Operate, overload, and roll back

Evidence boundary: runnable now versus design-only

• Runnable offline evidence: deterministic capstone outcomes, bounded step/tool counts, trace events, explicit no-match behavior, and regression comparison before and after a local change.
• Design-only with the provided package: admission service, durable queue, concurrent workers, dependency throttling, latency distribution, priced cost ledger, checkpoint migration, deployed canary, kill switch, and executable rollback. No load generator or release environment is included.

Use a synthetic workload table and rollback tabletop for the unimplemented controls. Label p50/p95/p99, queue age, throughput, and rollback results unmeasured unless you add an instrumented runner or deployment and actually observe them.

Choose the research or support capstone and produce an operations evidence pack:

1. define task classes and a service objective for terminal completion, not “AI uptime”;
2. map admission, queue, worker, model, tool, approval, state, and status boundaries;
3. write per-run, per-tenant, and global concurrency/budget limits;
4. create a load table with normal, burst, throttle, slow-tool, and approval-backlog cases;
5. record the available success, safety, trace, and cost-proxy evidence; define p50/p95/p99 and queue-age collection, but mark them unmeasured in the provided lab;
6. define idempotency and cancellation for every effect;
7. version the whole harness and write canary stop conditions;
8. conduct a rollback tabletop covering one in-flight run and one changed checkpoint schema; call it a rehearsal only if you add executable release and checkpoint state;
9. teach the design to a peer in five minutes without slides; record their two hardest questions.

Done when with the provided lab: the capstone tests pass, every metric and control is labeled observed, derived, or design-only, and the written gate would reject a documented bad candidate. Do not claim enforced canary blocking, overload behavior, operator recovery, or rollback until an added runtime demonstrates them.

Pause & Recall

1. Why can adding workers increase failure and latency?
2. What is the difference between a provider background request and durable agent run state?
3. Why is p95 more informative than an average but still insufficient alone?
4. Which denominator makes cost meaningful for an agent product?
5. Name five artifacts that must share a release identity.
6. What must rollback decide about in-flight runs?

Production lens

Dashboards should answer an operator's questions: Are users succeeding? Which task class is degraded? Is work waiting or executing? Which dependency is saturated? Are safety or approval patterns changing? What did this run cost? Which release caused the change? Can we stop effects without losing read-only service?

Alert on symptoms that matter—objective burn, oldest-job age, approval anomalies, policy denials, cross-tenant violations, trace loss, cost spikes—not every noisy internal fluctuation. Every page needs an owner, severity, runbook, safe mitigation, and evidence-preservation step. Practice provider outage, quota exhaustion, poisoned retrieval, stuck approval, checkpoint corruption, and model rollback as game days.

Chapter compression

• Separate admission from execution and use queues as bounded control surfaces.
• Scale against the constrained dependency, with backpressure and tenant fairness.
• Measure latency as a segmented distribution across the actual critical path.
• Optimize cost per successful outcome, not price per model call.
• Version and release the full harness through evidence gates.
• Rollback must reconcile state and effects, not merely old code.

Memory hook: Admit deliberately; execute durably; measure outcomes; release the harness; rehearse reversal.

Retrieval deck

• Q: What belongs in admission before an agent run is queued?
  A: Authentication, authorization, validation, tenant limits, an idempotency key, work estimate, and an accept/queue/reject decision.
• Q: Which latency number should operators use?
  A: Segmented end-to-end distributions plus queue and span timing; no single number is sufficient.
• Q: What is the honest cost metric for optimization?
  A: Total operating cost per outcome that passes the defined task criterion.
• Q: What is the unit of an agent release?
  A: The versioned harness: code, model, prompts, tools, policy, data/index, state schema, budgets, configuration, and evals.

Spaced review

• Now: draw the admission and execution planes from memory.
• +1 day: reconstruct the latency equation and name one optimization for each term.
• +3 days: write canary stop conditions for a model and tool-schema change.
• +7 days: inject a throttle or slow-tool failure and explain the backpressure response.
• +14 days: lead a rollback rehearsal and update the runbook from what surprised you.

Sources and further study

• Google SRE Workbook, Canarying Releases
• Kubernetes, Horizontal Pod Autoscaling
• OpenAI, Latency Optimization
• OpenAI, Cost Optimization
• OpenAI, Rate Limits
• OpenAI, Background Mode
• OpenTelemetry, Traces