Chapter 9.2 · Data, Models, Evaluation, Human Oversight, and Deployment Gates

Mission

By the end of this chapter, you can design a claim-based AI evaluation, a feasible human-oversight control, and a deployment gate whose scope matches its evidence.

Measurable outcome: Design an evaluation and deployment gate that states intended use, datasets, measures, slices, human oversight, thresholds, limitations, and post-deployment tests; score at least 3 of 4.
Prerequisites: Chapters 9, 12, 20, 25, 28, and 33.
Work product: An Evaluation and Deployment Gate Matrix.
Time: 85–110 minutes.

Before you read: Predict → Commit → Connect

Helios scores 91% on a 500-question test set. Should it enter pilot? Commit to yes, no, or “insufficient evidence.” List three facts hidden by the score.

Connect to Chapter 12: verification establishes specified behavior; validation establishes fitness for intended use. A single aggregate score rarely establishes either across a real workflow.

Evaluation is a system of claims and evidence

Start with claims, not a favorite metric. “Helios is accurate” is too broad. Better claims include: it retrieves evidence from the correct tenant; cites support for billing assertions; refuses unsupported account changes; routes out-of-scope cases to a human within the promised time; and does not expose sensitive data in logs.

For each claim define:

intended population and context;
harm if the claim fails;
test method and dataset provenance;
measure and threshold with rationale;
important slices and edge conditions;
assessor and independence needed;
limitations and residual uncertainty;
deployment and monitoring decision.

flowchart TB USE["Intended use, people, and harms"] --> CLAIM["Specific system claims"] CLAIM --> DATA["Test data, cases, provenance, and slices"] DATA --> TEST["Offline, adversarial, workflow, and operational tests"] TEST --> RESULT["Results, uncertainty, and limitations"] RESULT --> GATE["Pilot or deployment disposition"] GATE --> MON["Production monitoring and re-evaluation"]

Offline evaluation is valuable because it is repeatable and can probe dangerous cases without exposing users. It may not reproduce real prompts, changing data, latency, tool failures, human behavior, or distribution shift. Workflow evaluation observes people using the complete system. Production evaluation observes real outcomes, but must not become uncontrolled experimentation. Use progressive exposure with protections and consent/notice appropriate to the context.

Data is part of the claim

Record where evaluation cases came from, who can be represented, how labels were created, which version was used, and whether training or tuning data overlaps. Protect sensitive examples. Synthetic cases can target rare risks but may not represent natural language and workflow complexity. Historical data can reproduce past bias or obsolete policy. A strong suite combines sources and documents the gaps.

Slice results by meaningful conditions: language, region, customer type, intent, data availability, complexity, adversarial input, accessibility need, or consequence. Do not search slices merely to find a flattering result. Select them from context and risk mapping, and treat very small samples with appropriate uncertainty.

Model measures are not business outcomes. Precision, recall, groundedness, calibration, or task success may support a claim, but the program also needs handoff time, correction effort, incident signals, user understanding, and downstream effect. Cost and latency matter when they change availability or human behavior.

Human oversight must be an operating design

“A human reviews it” is not a control specification. Define:

which outputs or actions require review;
reviewer qualification and independence;
information shown, including source and uncertainty;
authority to reject, edit, stop, or escalate;
time available and workload;
automation-bias countermeasures;
what happens when no reviewer is available;
how review quality is sampled and improved.

flowchart LR AI["AI recommendation or proposed action"] --> RISK{"Risk tier and confidence evidence"} RISK -- "Low and bounded" --> AUTO["Allowed bounded response"] RISK -- "Review required" --> HUMAN["Qualified human with source and authority"] RISK -- "Prohibited or unsupported" --> BLOCK["Block and escalate"] HUMAN --> DEC{"Approve, modify, reject, or stop"} DEC --> LOG["Decision evidence and feedback"] AUTO --> LOG LOG --> MON["Monitor drift, harm, and workload"]

Human oversight can fail through volume. If one reviewer receives hundreds of plausible outputs per hour, formal approval may be a rubber stamp. Measure review load, disagreement, correction, missed defects, and fallback capacity.

Deployment gates connect evidence to exposure

Use multiple gates: sandbox, internal workflow, bounded pilot, limited production, broader production, and material-change review. A gate should state what the evidence supports and does not support. Example: evaluation on English billing questions does not support autonomous refunds, Spanish support, or fraud disputes.

Set stop conditions for severe events even when aggregate thresholds remain satisfied. Establish model/prompt/data/tool versions, evaluation suite version, monitoring baseline, rollback/disable path, and decision authority. Re-evaluate after material changes or new harm evidence.

Decision rights: Who owns what?

Product/domain owner owns intended use and acceptable user workflow.
Data/ML/Engineering owns dataset and model/system implementation evidence.
Evaluation/Quality teams own test-method integrity where separately chartered.
Security, Privacy, Legal, Compliance, Safety/Risk own specialist requirements and gates.
Human operations leaders own reviewer staffing, qualification, authority, and workload feasibility.
The TPM integrates claims, evidence, slices, limitations, gate criteria, exposure, and re-evaluation triggers.
The authorized release/risk forum accepts residual risk; aggregate performance does not decide automatically.

I do

I decompose Helios’s 91% score. I ask for dataset source and date, intent distribution, tenant-isolation tests, unsupported-answer behavior, sensitive-data handling, language slices, disagreement among labelers, human handoff, latency, and severe-case results. I recommend no pilot disposition until these claims and limitations are visible; I do not demand a mythical perfect score.

We do

The system meets its overall target but fails 3 of 20 rare account-takeover cases by suggesting unsafe recovery steps. Product argues that the sample is too small to block the pilot.

Together decide how severity and uncertainty should change the gate.

Show the model answer

Model answer

Do not average the rare high-consequence cases into the overall score. Block automated guidance for that intent, route it to qualified humans, expand targeted adversarial evaluation, and investigate the control failure. A pilot of other bounded intents may proceed if routing reliably prevents exposure and the release forum accepts the residual risk. Small sample size increases uncertainty; it does not convert observed severe failures into safety evidence.

Rubric (0–4)

0: Uses aggregate pass/fail only.
1: Notes severity but offers no scoped decision.
2: Adds testing or human review without proving routing/control feasibility.
3: Separates the intent, bounds exposure, strengthens evidence, and names authority.
4: Also tests routing failure, reviewer capacity, monitoring, and re-entry criteria.

You do

Complete an Evaluation and Deployment Gate Matrix:

Claim/harm	Dataset/case provenance	Measure and slice	Threshold/rationale	Result/uncertainty	Human control	Production signal	Gate disposition
No cross-tenant retrieval	Constructed and red-team cases by tenant topology	Any disclosure is severe	Zero observed; defense in depth		Server-bound scope + human stop	Denied-scope and canary tests

Include at least one functional claim, one security/privacy claim, one human-workflow claim, and one operational claim. Add a “claims not established” section.

Pause & Recall

Explain verification versus validation from memory. Recall Chapter 20: what decision does each measure drive? Recall Chapter 25: what makes an exception legitimate? Recall Chapter 33: which material changes trigger re-evaluation?

Production lens

Version the suite and cases. Prevent evaluation answers from leaking into system configuration. Monitor data and behavior drift, but remember drift detection itself has limits. Preserve user-report and incident paths. Sample human decisions for quality without turning monitoring into ungoverned surveillance.

Workplace artifact: deployment gate recommendation

Scope requested: [population, intents, regions, permissions] Claims supported: [ ] Claims not supported: [ ] Critical evidence/versions: [ ] Human oversight and capacity: [ ] Stop conditions and disable path: [ ] Residual risk and authority: [ ] Recommendation/next gate: [ ]

Chapter compression

Evaluation begins with use-context claims and harm, not an aggregate score. Combine traceable datasets, meaningful slices, offline and workflow tests, operational evidence, and explicit limitations. Design human oversight as a staffed control. Gates bound what the evidence permits.

Retrieval deck

Q: Why is one aggregate score weak? A: It hides task definition, provenance, slices, severity, workflow, uncertainty, and controls.
Q: What makes human oversight real? A: Qualified people with evidence, authority, time, manageable load, fallback, and quality monitoring.
Q: Why can a small severe-case sample block scope? A: Observed high-consequence failure is material, while small size increases uncertainty rather than proving safety.
Q: What belongs in “claims not established”? A: Uses, populations, languages, actions, or conditions outside the evidence.
Q: When should re-evaluation occur? A: After material model, data, prompt, tool, policy, workflow, context, or risk changes.

Spaced review

Now: Write one claim, harm, measure, slice, and deployment decision.
+1 day: Recreate the evaluation evidence stack from memory.
+3 days: Define a human-oversight control with authority, time, load, and fallback.
+7 days: Replace one aggregate metric with two risk-based slices and limitations.
+14 days: Review production evidence for a claim that needs re-evaluation or narrower scope.

Data, Models, Evaluation, Human Oversight, and Deployment Gates