Evaluation-Driven Agent Development

Mission

By the end, you can build an agent evaluation that measures outcomes, tool use, arguments, observable trajectories, grounding, safety, latency, and cost; combine deterministic, human, and model graders; and make release decisions with holdouts and uncertainty.

Prerequisite: Chapter 6.1 and the offline lab’s capstones, traces, budgets, and evaluation helpers. Build artifact: a versioned evaluation specification and promotion gate for the support capstone. Time: 85-105 minutes.

Before you read: Predict → Commit → Connect

Version B completes 86 of 100 cases; version A completes 82. B costs 60% more and violates refund policy once. Which version ships?

Commit to a decision. Write what you still need to know about the cases, graders, repeated runs, uncertainty, safety severity, latency, and business context.

An eval is an executable claim about acceptable behavior

An agent evaluation supplies a controlled task and environment, runs a versioned system, records observable behavior, applies grading rules, and aggregates results for a decision.

The evaluated system is not only the model. It includes instructions, context assembly, retrieval, tools, schemas, policies, approvals, memory, orchestration, model settings, runtime, and environment. Record all of them. A model score from a different harness does not predict your application reliably.

Write the release claim first: “For eligible synthetic duplicate-charge tickets, the candidate proposes no more than the documented amount, never executes without approval, cites the policy, completes within the tool budget, and meets latency and cost targets.” The dataset and graders must test that claim.

Measure the complete observable behavior

Use a scorecard with at least these dimensions:

Trajectory means the inspectable sequence of model requests, proposed actions, validations, tool calls, observations, approvals, state transitions, and final output. It does not mean hidden chain-of-thought. Do not make access to private reasoning a prerequisite for debugging or evaluation.

A single composite score can hide a severe failure. Keep dimensions separate, define hard vetoes, and show slice results. “Average 91” is meaningless if privacy deletion cases fail.

Build cases from the task distribution and failure history

Begin with a task taxonomy: common paths, difficult paths, high-impact paths, boundary values, languages, ambiguity, dependency failure, adversarial content, and accessibility needs. Sample according to expected traffic, but oversample rare catastrophic risks and report both weighted and unweighted views.

Each case should include:

• stable ID, purpose, risk tier, and slice labels;
• synthetic or authorized minimized input data;
• reset procedure and initial state;
• allowed tools and relevant policy version;
• expected outcomes plus acceptable alternatives;
• forbidden outcomes and critical invariants;
• grading method, evidence, and owner;
• budgets for attempts, tool calls, latency, and cost.

Turn production failures into sanitized regression cases. Add neighboring variants so the system cannot memorize one string. Keep ambiguous cases; mark them for human judgment or clarification behavior instead of inventing a single false gold answer.

Development sets and sealed holdouts

Separate data by decision purpose:

• development set: run frequently while changing the system;
• validation set: compare candidates and tune thresholds;
• sealed holdout: run only for promotion or scheduled audits;
• online monitoring set: sampled, privacy-controlled production-like behavior after release.

Deduplicate semantically, not only by exact text. Keep related templates, users, documents, and incidents in one split. Restrict holdout access and log use. If a holdout leaks into prompts, examples, retrieval corpora, model fine-tuning, or repeated manual tuning, retire and replace it.

The holdout is not sacred forever. Task distributions and attacks change. Version the evaluation and preserve old anchors so improvement on new cases does not erase regression history.

Three grader families

Deterministic graders

Code can check schema, exact state, database records, tool allowlists, argument bounds, required trace events, citations, budgets, security invariants, tests, and file diffs.

They are fast, repeatable, and auditable. They can also be brittle or incomplete: an exact string check may reject a correct paraphrase, and a final-state check may miss a dangerous detour. Prefer semantic business state over surface text when possible.

Human graders

Domain experts can judge usefulness, nuance, truthful uncertainty, tone, and multiple acceptable solutions. Humans are essential for calibrating ambiguous graders and high-impact cases.

They are slower and costly, and they disagree. Use a clear rubric with examples, blind candidate identity, random order, training and calibration items, independent double review for critical cases, disagreement resolution, and inter-rater agreement reporting. Human labels are measurements with uncertainty, not unquestionable truth.

Model graders

A model grader can scale rubric-based judgments such as groundedness, completeness, or policy interpretation. It is useful when deterministic checks cannot capture meaning and human review cannot cover every run.

It can be biased by position, verbosity, style, model family, or prompt injection inside the evaluated output. It may share the candidate’s blind spots, produce inconsistent labels, or confidently misapply a rubric. Never make an uncalibrated model grader the sole authority for high-impact safety or release decisions.

Mitigate these limits by using narrow criteria, structured outputs, evidence citations, blinded candidates, order swaps for pairwise judgments, repeated grading where justified, an abstain path, and continuous comparison against an expert-labeled calibration set. Prefer deterministic state checks whenever the underlying fact is machine-verifiable.

Groundedness needs claim-level evidence

Citation presence is not grounding. A valid evaluation asks:

1. Does the cited source exist in the allowed evidence set?
2. Does the cited span support the specific claim?
3. Is the claim faithful to scope, date, population, and uncertainty?
4. Are important claims unsupported even if minor claims have citations?
5. Did the agent rely on evidence that was unavailable at decision time?

Use deterministic checks for source identity and access, then human or calibrated model judgment for entailment when necessary. Seed contradiction and “no answer” cases. Reward abstention or clarification when evidence is insufficient.

Reliability requires repeated trials and uncertainty

Agent runs can vary because of model sampling, retrieval ties, concurrency, or external dependencies. One pass per case estimates possibility, not reliability.

For important stochastic cases, run repeated independent trials with the environment reset. Report numerator and denominator, not just a percentage. Include a confidence interval appropriate to the unit of sampling. When cases have repeated runs, resample or aggregate at the case level so hundreds of repeats on five cases do not masquerade as broad coverage.

Compare candidates on the same cases and environmental conditions. A paired comparison often separates version effects from case difficulty. Show the distribution of latency and cost, not only the mean. Tail behavior matters for multi-step systems.

Do not declare a winner because 86% exceeds 82%. Ask whether the interval is narrow enough, the improvement appears across critical slices, graders are reliable, and hard safety constraints remain intact.

Promotion gates, not leaderboard theater

Define gates before running the holdout. For example:

• zero critical cross-tenant, secret-disclosure, unauthorized-write, or false-execution violations;
• lower confidence bound for task success above the current production floor;
• no material regression on any high-risk slice;
• tool and argument correctness above threshold;
• groundedness above threshold with calibrated grader agreement;
• p95 latency and cost per successful task within budgets;
• all known severity-one regressions fixed;
• canary and rollback plan approved.

A safety veto can outweigh aggregate success. If trade-offs are permitted, the accountable product and risk owners—not a model grader—set them explicitly.

Evaluation-driven development means failures choose the next engineering work. Group failures by root layer—context, route, schema, authorization, tool, recovery, grounding, or grader—then make the smallest controlled change and rerun all affected regressions.

I do: audit the offline eval helper

Run its tests:

cd courses/ai-agents/labs
python3 -m unittest tests.test_evals -v

EvalCase and evaluate_cases deliberately provide a small deterministic foundation. They check expected status, required or forbidden output strings, maximum tool calls, required trace kinds, and runner failures. The report preserves per-check evidence instead of only a score.

That is useful but incomplete. It does not yet grade exact tool arguments, final business state, claim-level grounding, policy severity, latency, or cost. It does not repeat stochastic runs or compute uncertainty. The correct response is to extend the evaluation contract for the real task, not to mistake a passing unit test for production readiness.

We do: grade the refund case

For synthetic ticket T-1001, define a layered rubric:

Run the denial and tool-name-allowlisted simulation variants. Passing one does not cover the other, and the current allowlist does not prove exact authenticated human approval; record that as an evaluation gap.

You do: LAB-17 — Create a promotion-quality eval specification

Run the capstone and core regressions:

cd courses/ai-agents/labs
python3 -m unittest \
  tests.test_capstones \
  tests.test_evals \
  tests.test_agent.AgentLoopTests.test_tool_call_budget_stops_excess_calls \
  tests.test_memory_approvals_trace.TraceTests.test_trace_recursively_redacts_sensitive_keys -v

Design at least twelve cases across research and support, including no-match evidence, password-reset secret handling, refund denial and approval, privacy escalation, unknown ticket, malformed arguments, transient failure, prompt injection, and budget exhaustion.

For every case, specify all eight dimensions, grader family, hard veto, and expected evidence. Split cases into development, validation, and sealed holdout families without near duplicates. Define repeated-run counts for stochastic behavior, the interval and comparison method, and the exact promotion decision.

Done when: a reviewer can tell what a score means, which risks it does not cover, how graders were calibrated, and why the proposed gate is statistically and operationally defensible.

Pause & Recall

1. Name the eight required evaluation dimensions.
2. What is observable trajectory quality?
3. Give one strength and one limitation of each grader family.
4. Why keep a sealed holdout?
5. Why are repeated runs on a few cases not broad evidence?

Production lens

Run a small deterministic suite on every change, broader suites on scheduled or release boundaries, sealed holdouts for promotion, and privacy-controlled online evaluation after deployment. Cache only when inputs and versioned dependencies truly match; otherwise caching can conceal variation.

Slice results by task, risk, language, tool, tenant class, route, model, and version while preserving privacy. Watch for evaluator drift as policies and graders change. Version human rubrics and model-grader prompts. Recalibrate model graders against fresh expert labels and investigate disagreement rather than averaging it away.

Store enough sanitized evidence to reproduce scores. Protect holdout data, customer-derived cases, and telemetry from unauthorized training or prompt exposure. Evaluation infrastructure is itself production software: access-control it, test it, observe it, and review its supply chain.

Chapter compression

• Evaluate the complete agent system and record every relevant version.
• Measure task, tool, arguments, trajectory, grounding, safety, latency, and cost separately.
• Combine deterministic, human, and calibrated model graders according to their strengths.
• Protect sealed holdouts from tuning and semantic leakage.
• Repeat stochastic trials and report uncertainty at the correct sampling unit.
• Predefine hard safety vetoes and promotion gates before viewing holdout results.

Memory hook: Test the whole path; grade by evidence; hold out; quantify doubt.

Retrieval deck

• Q: What eight dimensions should an agent eval cover?
  A: Task success, tool selection, argument correctness, observable trajectory, groundedness, safety, latency, and cost.
• Q: When is a deterministic grader preferable?
  A: When schema, state, arguments, receipts, tests, or policy invariants can be verified directly by code.
• Q: Why can a model grader not be treated as ground truth?
  A: It can be biased, inconsistent, injected, correlated with the candidate, or wrong; calibrate it against experts and allow abstention.
• Q: What does a holdout protect?
  A: An estimate of generalization that has not been repeatedly optimized through prompts, examples, retrieval, or manual tuning.

Spaced review

• Now: write one grader for each of the eight dimensions.
• +1 day: reconstruct the grader strengths and failure modes.
• +3 days: split a case family without semantic leakage.
• +7 days: compare two candidates with paired repeated trials and intervals.
• +14 days: audit whether a release gate could hide a critical slice behind an average.

Sources and further study

• NIST: AI Test, Evaluation, Validation, and Verification
• Yao et al.: τ-bench—Tool-Agent-User Interaction in Real-World Domains
• Liu et al.: AgentBench—Evaluating LLMs as Agents
• Anthropic: Demystifying Evals for AI Agents
• Shi et al.: Judging the Judges—Position Bias in LLM-as-a-Judge