Specify Before You Build—Goal, Contract, and Baseline Eval

Reliability starts before the first model call. Specify a bounded job, define evidence of success, and record the simplest baseline so every later improvement has something honest to beat.

Mission

By the end, you can turn an agent idea into a falsifiable task contract, define the simplest non-agent baseline, and run a small evaluation that exposes failure before implementation choices hide it.

Prerequisites: Modules 1-3. Build artifact: a versioned task contract, baseline decision, and first evaluation report. Time: 80-100 minutes.

Before you read: Predict → Commit → Connect

A team proposes: “Build an autonomous research agent that finds the truth about any topic and writes a perfect report.”

Write three reasons this cannot be evaluated as stated. Then rewrite only the word perfect into a measurable phrase. Notice that one word is not enough to repair the rest of the specification.

A specification is a testable promise

A useful task specification tells a builder, evaluator, operator, and affected user what the system is allowed to attempt and what counts as a safe outcome.

“Research any topic” is an aspiration. This is closer to a contract:

Given a question about the four-document synthetic safety corpus, return a concise answer supported by valid document IDs, or state that no matching evidence was found. Use only read-only corpus tools. Stop within five steps and five tool calls. Never use external knowledge or invent a citation.

Someone can now construct cases that prove the promise false.

The loop begins with the contract, not with prompt tuning. If the contract changes, version it and explain why.

The twelve-part task contract

Use this structure:

1. User and job: who needs what outcome, in which setting?
2. Goal: one observable outcome for one run.
3. Inputs: required fields, accepted formats, size limits, and data classifications.
4. Outputs: structure, audience, evidence, uncertainty, and maximum size.
5. In-scope decisions: which judgments may the model propose?
6. Tools and data: exact eligible actions and approved sources.
7. Permissions: actor, tenant, resource, scopes, and credential owner.
8. Prohibitions: actions, sources, disclosures, and claims that must never occur.
9. Approvals: which exact effects pause, who can decide, and expiry behavior.
10. Budgets and stop states: steps, calls, time, cost, retries, escalation, failure.
11. Success and failure evidence: machine checks, reviewer rubric, and critical errors.
12. Owner and lifecycle: accountable team, release gate, monitoring, rollback, and retirement.

The contract should fit on one or two pages. Detailed schemas, rubrics, and datasets can be linked artifacts.

Separate outcome, behavior, and operation

These are different targets:

• Outcome: the answer is supported and useful.
• Behavior: only allowed tools were used; no citation was invented.
• Operation: the run stayed within latency, cost, and availability limits.

A system can produce a correct answer through forbidden data access. It can behave safely yet fail to answer. It can do both but cost too much to operate. Score all three.

Write completion evidence before “done”

Avoid “the model decides when finished.” Define task-specific completion evidence.

For a research brief:

• every material claim maps to a document ID returned in this run;
• cited excerpts support the claims;
• the source belongs to the authorized corpus and correct version;
• conflicting evidence is disclosed;
• unsupported questions receive an abstention;
• the result satisfies length and structure limits;
• no write tool or external source was used;
• trace and budget records are complete.

Completion is a validated state, not a sentence containing “done.”

Establish the least-complex baseline

Before an agent, test:

1. human process with better search;
2. deterministic rules;
3. one model call with supplied context;
4. fixed retrieve-read-answer workflow;
5. workflow with one clarification branch;
6. bounded agent only if adaptation adds measurable value.

Record baseline quality, critical errors, latency, cost, and reviewer time. The agent must beat the baseline on the business outcome enough to pay its reliability tax.

Do not choose a deliberately weak baseline. A strong fixed workflow is useful even if it “wins”—you have discovered a simpler product.

Define the agent advantage as a hypothesis

Example:

When the first search lacks enough evidence, one bounded query revision will raise evidence recall on difficult cases by at least 10 percentage points without increasing critical citation errors, with no more than one additional tool call at the 95th percentile.

This can fail. That is what makes it useful.

Build the evaluation set from the risk model

Start small but cover distinct failure modes:

One aggregate score can hide catastrophe. Track critical failures separately:

• cross-tenant disclosure;
• unauthorized or unapproved side effect;
• fabricated evidence;
• dangerous instruction followed from untrusted content;
• silent success after an unknown transaction state.

A single confirmed critical error may block release even when average task success is high.

Choose checks by what can be known

Use the strongest available oracle:

Deterministic checks

Best for exact properties:

• schema and type validity;
• source ID exists;
• no forbidden tool ran;
• approval event precedes a write;
• budgets were respected;
• tenant IDs never crossed;
• calculation matches reference code.

Reference comparisons

Useful when expected labels or outcomes exist. Allow multiple correct formulations; compare facts or structure rather than exact prose when wording is not the requirement.

Human rubrics

Use trained reviewers for relevance, clarity, nuanced support, and high-stakes judgment. Define dimensions and anchored examples. Measure reviewer agreement and adjudicate disagreements.

Model graders

They can scale subjective checks, but must be calibrated against human labels and protected from seeing irrelevant metadata or a preferred-system identity. Prefer pairwise or criteria-based judgments over vague “score 1-10.” Never let an unvalidated model grader be the only gate for a critical safety property.

Freeze a baseline before optimization

For every run, capture:

• case and dataset version;
• system, prompt, model, tool, policy, and corpus versions;
• final status and output;
• tool calls, approvals, errors, and terminal reason;
• deterministic checks and grader results;
• latency and resource use;
• failure category and reviewer notes.

Run repeated trials where model variability matters. Report counts and uncertainty, not a single lucky run. Keep a development set for iteration, a held-out set for release decisions, and production-derived cases with privacy protection.

Do not put held-out answers into prompts, demonstrations, or skill references. Evaluation leakage turns the test into training material.

I do: specify the offline research agent

Here is a compact contract for agent_lab.capstones.research:

The implementation is actually closer to a fixed workflow expressed through an agent loop than an open-ended agent. That makes it a strong first baseline.

We do: turn one promise into cases

Promise: “Return no-match when the corpus lacks relevant evidence.”

Create at least four cases:

1. nonsense terms with no overlap;
2. a plausible topic absent from the corpus but containing common words;
3. a question with one exact rare term from a document;
4. a hostile request to invent an answer.

Expected behavior differs. Case 1 tests an easy lexical miss. Case 2 exposes false-positive relevance. Case 3 tests recall. Case 4 tests the source boundary. One happy-path question cannot represent the promise.

You do: LAB-10 — Freeze the baseline

Run this report before changing any implementation:

PYTHONPATH=courses/ai-agents/labs python3 - <<'PY'
from agent_lab.capstones.research import run_research
from agent_lab.evals import EvalCase, evaluate_cases

cases = (
    EvalCase(
        "direct-evidence",
        "How should a safe AI agent be evaluated?",
        expected_contains=("[R-02]", "Sources:"),
        max_tool_calls=3,
    ),
    EvalCase(
        "memory-evidence",
        "How should session memory be isolated?",
        expected_contains=("[R-03]",),
        max_tool_calls=3,
    ),
    EvalCase(
        "obvious-no-match",
        "quasar nebula zygote",
        expected_contains=("No matching evidence",),
        forbidden_contains=("Sources:",),
        max_tool_calls=1,
    ),
    EvalCase(
        "plausible-no-match",
        "What does the corpus say about quantum gravity?",
        expected_contains=("No matching evidence",),
        forbidden_contains=("Sources:",),
        max_tool_calls=1,
    ),
)

report = evaluate_cases(
    cases,
    lambda case: run_research(case.user_input, run_id="baseline-" + case.case_id),
)
for result in report.results:
    print("PASS" if result.passed else "FAIL", result.case_id)
    for check in result.checks:
        if not check.passed:
            print("  -", check.name, check.detail)
print(f"baseline={report.passed}/{report.total} ({report.pass_rate:.0%})")
PY

Expect the plausible no-match case to expose the toy retriever’s false-positive behavior. Keep that failure in the baseline report.

Now create your artifact:

1. Write the twelve-part contract for your chosen task.
2. Name the simplest baseline and the hypothesis an agent must beat.
3. Create at least 12 cases across six case families.
4. Define deterministic checks before subjective rubrics.
5. Mark critical-error vetoes and minimum release thresholds.
6. Record versions and the baseline result, including failures.
7. Ask a reviewer to write three cases you did not anticipate.

Done when: a new builder can implement a competing system and a skeptical evaluator can determine whether it met the same promise.

Pause & Recall

1. What twelve fields belong in a task contract?
2. Why must the baseline be the least-complex credible solution?
3. How do outcome, behavior, and operational checks differ?
4. Why can a high average score fail a release gate?
5. Which properties deserve deterministic checks?
6. What is evaluation leakage?

Production lens

Treat the contract, dataset, graders, and thresholds as versioned release artifacts. Require an owner for every critical case and a documented response when it fails. Do not delete hard failures because they lower a dashboard score.

Sample production traces for new failure modes, then de-identify and review them before adding cases. Monitor data drift, source changes, tool errors, reviewer disagreement, abstention, and cost. A release gate is not permanent proof; it is evidence under a named distribution and configuration.

When the product expands scope, do not merely add tools. Rewrite the contract, threat model, cases, approval rules, and rollback plan first.

Chapter compression

• A useful agent specification is a falsifiable promise with explicit authority and stop boundaries.
• Define completion evidence before prompts or frameworks.
• Establish the strongest simple baseline and state the agent advantage as a hypothesis.
• Cover outcomes, behavior, security, failures, and operations with representative cases.
• Use deterministic checks where possible and calibrate subjective graders.
• Preserve baseline failures; they guide the build and prevent self-deception.

Memory hook: Promise, baseline, cases, evidence, gate.

Retrieval deck

• Q: What makes a goal evaluable?
  A: Observable inputs, outputs, boundaries, completion evidence, failure states, and thresholds.
• Q: Why build a non-agent baseline first?
  A: To learn whether adaptation adds enough measurable value to justify added variability, cost, and risk.
• Q: What is a critical-error veto?
  A: A failure category severe enough to block release regardless of the aggregate score.
• Q: What must every evaluation report identify?
  A: Dataset, system, model, prompt, tool, policy, corpus, grader, and run configuration versions.

Spaced review

• Now: write the one-paragraph contract for the research lab from memory.
• +1 day: list twelve case families or contract fields without looking.
• +3 days: create one fluent-but-unsafe case that average accuracy would hide.
• +7 days: compare your baseline and agent hypothesis using failure economics.
• +14 days: rerun the frozen baseline after one deliberate change and explain the delta.

Sources and further study

• Anthropic, Demystifying evals for AI agents
• OpenAI, Evaluation best practices
• NIST, AI Risk Management Framework
• Breck et al., The ML Test Score: A Rubric for ML Production Readiness