Observability, Debugging, and Incident Response

Mission

By the end, you can design privacy-aware traces, logs, metrics, and service-level objectives; debug an agent from observable evidence; release through canaries with rollback; and preserve an incident record that supports containment and learning.

Prerequisite: Chapters 6.1-6.2 and the offline lab’s structured trace events. Build artifact: an observability contract, SLO sheet, and simulated incident packet for the support capstone. Time: 85-105 minutes.

Before you read: Predict → Commit → Connect

After a model update, p95 latency doubles and several users receive “refund completed,” but the payment dashboard has no matching receipts.

Write the first three actions you would take. Then list the evidence needed to distinguish model delay, tool retries, a trace bug, a false final claim, a missing receipt, and a real payment incident.

Observability turns execution into evidence

Agent observability should answer:

• What request and version set produced this run?
• Which observable decisions, validations, tool calls, approvals, and state transitions occurred?
• Where did time, attempts, and cost accumulate?
• Which evidence supported the final claims?
• Did a side effect occur, fail, or remain unknown?
• Which users, tenants, systems, and releases may be affected?

Use complementary signals:

• Trace: the causal path of one run, represented as related spans or events.
• Log: a discrete structured event useful for audit, security, or local diagnosis.
• Metric: an aggregation over many events, used for rates, distributions, saturation, and SLOs.
• Profile: sampled resource use when code-level CPU or memory diagnosis is needed.

More telemetry is not automatically better. Full prompts, messages, screenshots, tool results, or model reasoning can contain personal data, secrets, copyrighted material, and hostile content. Collect the minimum evidence required for a declared operational purpose.

The run trace is the spine

Give every run a stable trace_id or run_id. Give each operation a span_id and parent. Propagate correlation through tool adapters, durable jobs, remote agents, and approval systems without treating it as authentication.

A practical agent span records:

• operation kind and sanitized name;
• start, end, duration, status, and typed error class;
• parent, run, task, and attempt identifiers;
• tenant-safe or pseudonymous scope, never a raw secret;
• model, prompt, policy, schema, tool, and orchestration versions;
• model request timing and usage counts, where available;
• tool name, argument schema version, and safe argument fingerprint;
• validation and authorization outcome;
• approval reference, actor class, scope, and expiry—not credentials;
• idempotency and receipt references for effects;
• budget before/after, retry count, and stop reason;
• cited source IDs and output status.

Do not store hidden chain-of-thought as the debugging plan. Observable proposals, tool arguments, policy decisions, state transitions, and receipts are more reliable operational artifacts.

Real distributed traces use temporal parent-child relationships; the diagram emphasizes the evidence attached to each operation. Keep span names low-cardinality. Put run-specific values in attributes, not metric or span-name dimensions.

Redaction is a pipeline, not a regular expression

Create a telemetry data classification before instrumentation. For each field, choose: do not collect, transform, encrypt and restrict, sample, or retain briefly.

Usually exclude or protect:

• passwords, one-time codes, access tokens, cookies, API keys, private keys, and connection strings;
• raw payment, health, identity, or highly sensitive personal data;
• full prompts, messages, retrieved documents, screenshots, and generated files by default;
• unrestricted tool arguments and results;
• secrets embedded in values whose key looks harmless.

Layer controls:

1. emit structured allowlisted fields at the source;
2. redact recursively by key and known value patterns;
3. replace account and session identifiers with scoped pseudonyms;
4. truncate depth, string size, arrays, and artifact previews;
5. scan for secret canaries and high-risk data classes;
6. encrypt in transit and at rest;
7. restrict access by operational role and record access;
8. enforce retention, deletion, export, and regional requirements;
9. test collectors, exporters, dashboards, alerts, and backups—not only application code.

Redaction can remove debugging detail. Preserve a secure, separately authorized path for necessary incident evidence rather than weakening every production log. Never make ordinary dashboard access a back door to customer content.

Debug from the outside in

Use a repeatable sequence:

1. identify affected SLI, task slice, time window, tenant scope, and first bad version;
2. compare candidate and control metrics;
3. select representative failed and successful run IDs;
4. inspect trace structure for the first divergent operation;
5. verify state and receipts in the authoritative system, not the final prose;
6. reproduce with the recorded version set and synthetic or authorized minimized input;
7. isolate the layer: context, model proposal, validation, authorization, tool, dependency, durable state, or output contract;
8. add a regression and verify the fix across adjacent slices.

A trace shows correlation and order, not necessarily causation. A slow model span may be waiting on upstream quota; a tool error may be a symptom of a bad argument. Compare evidence and test hypotheses.

Service levels for useful, safe work

A service-level indicator (SLI) is a measured behavior. A service-level objective (SLO) is a target for an SLI over a window. The error budget is the tolerated amount of performance outside the target.

Agent SLOs need more than HTTP availability:

Define the numerator, denominator, exclusions, window, source of truth, delay, and owner. Slice by task and risk. An overall SLO can look healthy while privacy or one language fails.

Safety is not reduced to “99.9% authorized.” Some outcomes—cross-tenant disclosure, unapproved payment, secret exfiltration—may have zero tolerance and trigger immediate containment even when a statistical SLO remains green.

Alert on user symptoms and fast/slow error-budget burn, not every individual model error. Page when immediate human action can reduce harm. Route lower urgency trends to tickets and evaluation work.

Canary, compare, and roll back

Version and independently release model, prompt, context policy, retrieval, tool schema, business policy, grader, and runtime. Start risky changes in offline evals, then shadow or dry-run modes that cannot write. Move to a small canary with explicit eligible traffic, duration, guardrails, and owners.

Compare canary with a simultaneous control to reduce time-based confounding. Use absolute safety and SLO gates as well as relative metrics. A canary and control can both be unhealthy.

A rollback must be executable under pressure. Keep the previous known-good versions, compatibility with in-flight runs, a capability kill switch, and state reconciliation. Rolling back code does not undo a payment, message, deletion, or poisoned memory already produced.

Incident response for agent systems

Prepare before an incident:

• owners, severity rubric, on-call routes, and decision authority;
• capability-specific kill switches and credential-revocation procedures;
• runbooks for data leakage, unauthorized effects, false claims, loops, provider outage, and sandbox escape;
• protected evidence storage and retention rules;
• communication templates and legal, privacy, security, and vendor contacts;
• rehearsed restoration and reconciliation.

During an incident, prioritize people and containment. Pause the risky capability, restrict traffic, revoke exposed credentials, stop remote peers, or roll back the suspect version as appropriate. Preserve relevant state and telemetry; do not “clean up” evidence destructively. Verify authoritative systems and determine whether harm is continuing.

Create an incident evidence packet containing:

• incident ID, reporter, severity, owner, and timestamps;
• affected task, tenant scope, users, systems, and data classes;
• first known bad and last known good versions;
• run/trace IDs and a sanitized event timeline;
• model, prompt, policy, schema, tool, dependency, and deployment versions;
• approvals, operation IDs, receipts, and system-of-record confirmation;
• metrics and dashboard snapshots with query definitions;
• containment and recovery actions with actor and time;
• evidence locations, access history, and integrity metadata;
• hypotheses clearly separated from confirmed facts;
• customer or regulator communication decisions where applicable;
• root control gap, regressions, owners, and follow-up dates.

Incident response is a cycle: prepare, detect, analyze, contain, recover, and improve. A post-incident review should be blameless toward people but precise about failed controls and ownership.

I do: read the lab’s trace contract

Run the trace tests:

cd courses/ai-agents/labs
python3 -m unittest tests.test_memory_approvals_trace.TraceTests -v

The lab trace is deliberately small. Each event has an index, kind, and immutable sanitized payload. Recursive processing redacts keys containing fragments such as authorization, password, secret, token, or api_key; bounds nested depth, collection length, and string length; and replaces unsupported objects with a type marker.

Those are useful controls, not a production completeness claim. Key-based redaction can miss a secret stored under note. A production design also needs source allowlists, value detection, access control, retention, exporter tests, version attributes, timestamps, parent spans, metrics, and evidence references.

We do: debug a false refund claim

Assume a run says “refund completed,” but the simulator receipt contains executed: false.

Together trace the first violated contract:

1. The proposed amount and approval may both be valid.
2. The tool result is authoritative about simulation status.
3. The final-output validator should require a real execution receipt before completion language.
4. The incident scope includes runs from the same output-contract version, not every refund request.
5. Containment can disable completion wording or the refund route while preserving read-only support.
6. The regression checks that executed: false, missing receipt, and outcome unknown never produce “completed.”

The model’s confidence or explanation does not override the receipt.

You do: LAB-18 — Instrument, set objectives, and rehearse response

Run the relevant evidence and failure tests:

cd courses/ai-agents/labs
python3 -m unittest \
  tests.test_memory_approvals_trace.TraceTests \
  tests.test_agent.AgentLoopTests.test_unexpected_handler_error_is_sanitized \
  tests.test_agent.AgentLoopTests.test_transient_tool_failure_retries_within_budget \
  tests.test_capstones.SupportCapstoneTests.test_refund_is_denied_by_default_and_never_executes -v
python3 -m agent_lab.capstones.support T-1001 --approve-refund

Evidence boundary: runnable now versus design-only

• Runnable offline evidence: ordered in-memory trace events, recursive key-based redaction and size bounds, sanitized unexpected errors, bounded transient retries, and a receipt showing that the approved simulation still has executed=false.
• Design-only with the provided package: timestamps and parent spans, telemetry export, value-based secret scanning, retained audit evidence, aggregated metrics, SLO calculation, live alerts, canary traffic, kill switches, and deployment rollback.

The SLO and incident work below is a tabletop using synthetic evidence. Do not claim that a real alert fired, a production capability was contained, or a deployed release was rolled back unless you add and exercise that environment.

Produce three artifacts:

1. Observability contract: trace hierarchy, event schema, version fields, redaction and retention, metric dimensions, access roles, and five sample queries.
2. SLO sheet: definitions and targets for task success, groundedness, safe action, p95 latency, reconciliation time, and cost per success; include hard safety invariants and alert ownership.
3. Incident packet: simulate a false-execution claim after a candidate release. Record timeline, affected versions, sanitized evidence, containment, rollback, receipt verification, root control gap, and regression.

Done when with the provided lab: the trace tests pass, the synthetic false claim is reconciled against executed=false, and the observability contract, SLO sheet, and incident packet clearly mark unimplemented telemetry and release controls. Live detection, containment, and rollback remain production-extension evidence, not offline-lab results.

Pause & Recall

1. What question does each signal—trace, log, metric, profile—answer?
2. Why is key-based redaction insufficient?
3. Define SLI, SLO, and error budget.
4. Why compare a canary with both a control and absolute gates?
5. What evidence distinguishes a proposed effect from a completed effect?

Production lens

Test telemetry loss, collector failure, backpressure, sampling, clock skew, trace propagation, and high-cardinality fields. Security and effect events may require unsampled or separately durable audit records, while routine model spans can be sampled. Document the difference between operational trace and regulated audit record.

Dashboards must link quality, reliability, safety, latency, and cost by version. Provide drill-down from an alert to sanitized representative run IDs, then to authoritative receipts under stricter access. Monitor missing telemetry itself; silence during an incident is not health.

Run game days for provider outage, runaway retries, leaked credential, remote-agent compromise, sandbox escape, false success claims, and corrupted durable state. Measure time to detect, contain, reconcile, restore, and add regression coverage. Verify that rollback and kill switches work while the primary dependency is unavailable.

Chapter compression

• Traces explain one run; logs record events; metrics summarize populations; profiles diagnose resource use.
• Instrument observable proposals, controls, calls, state, budgets, evidence, and receipts—not hidden chain-of-thought.
• Minimize telemetry and enforce redaction, access, retention, and exporter-level tests.
• Define SLOs for useful and safe task completion, not HTTP uptime alone.
• Canary against a control with hard gates and a rehearsed rollback or capability kill switch.
• Preserve an incident timeline, versions, approvals, receipts, scope, actions, and follow-up regressions.

Memory hook: Trace the path; protect the evidence; watch the objective; rehearse recovery.

Retrieval deck

• Q: What is the authoritative proof that a side effect completed?
  A: A verified system-of-record state or provider receipt tied to the stable operation ID—not the model’s final wording.
• Q: What belongs on an agent trace?
  A: Correlated operations, version set, timing, typed outcomes, validation and approval decisions, budgets, retries, evidence IDs, and receipt references with minimized payloads.
• Q: What makes a canary decision sound?
  A: Predefined duration and population, simultaneous control, absolute safety/SLO gates, representative slices, and an executable rollback.
• Q: What must happen to an incident finding?
  A: Containment and recovery, preserved evidence, root-control analysis, an owned fix, and a regression that blocks recurrence.

Spaced review

• Now: sketch a trace tree for one tool-using run.
• +1 day: reconstruct the telemetry minimization and redaction pipeline.
• +3 days: write one SLI precisely with numerator, denominator, window, exclusions, and owner.
• +7 days: run a canary rollback tabletop with an ambiguous side effect.
• +14 days: conduct an incident game day and verify the evidence packet and regression.

Sources and further study

• OpenTelemetry: Signals
• OpenTelemetry: Generative AI Semantic Conventions
• Google SRE Workbook: Implementing SLOs
• Google SRE Workbook: Canarying Releases
• NIST SP 800-61 Rev. 3: Incident Response Recommendations
• OWASP Logging Cheat Sheet