AI Eval Coverage Confidence Standard

What kinds of questions an AI evaluation must contain before its score can be trusted to mean anything. Declare a target mix for the application, classify every question once, and compare actual coverage with the target—by case type and by topic.

Suite composition

Every eval question receives exactly one primary case type. Primary types are mutually exclusive, so a suite’s composition always totals 100%. Other properties a case exercises are kept as secondary tags and never double-counted. When several types apply, the first in this precedence wins: FP challenge → FN challenge → Grounding → Insufficient → Negative → Boundary → Positive.

Positive / supported

Valid, answerable inputs with a verifiable expected answer.

Boundary

On, just below and just above meaningful thresholds, limits or transitions.

Negative / false premise

Invalid, prohibited or false-premise inputs with an expected rejection or correction.

Insufficient / conflicting info

Underspecified, ambiguous or contradictory inputs with a declared acceptable response.

Hallucination / grounding

Claims checked against supplied evidence, including missing or conflicting evidence.

False-negative challenge

A case that MUST trigger the positive decision even though it is disguised or subtle.

False-positive challenge

A case that must NOT trigger the task’s positive decision (benign lookalike).

False-positive and false-negative challenges are intended test cases. Measured false-positive and false-negative rates are outcomes with their own denominators and thresholds; report them separately, not as slices of the composition.

Target profiles

Targets depend on the application. These proposed starting profiles are configurable; set the mix before evaluating, then compare actual composition against it. A zero is no floor, not a ban.

Profile	Positive	Boundary	Negative	Insufficient	Grounding	FN challenge	FP challenge
General assistant / Q&A Balanced challenge set: supported answers plus false premises, limits, missing information and grounding.	Positive 30%	Boundary 15%	Negative 15%	Insufficient 15%	Grounding 15%	FN challenge 5%	FP challenge 5%
Retrieval / grounded answers Missing, contradictory, outdated and irrelevant evidence; unsupported answers and unnecessary refusals.	Positive 25%	Boundary 10%	Negative 10%	Insufficient 15%	Grounding 30%	FN challenge 5%	FP challenge 5%
Tool use / agent workflows Authorization, wrong-tool selection, partial failure and irreversible-action boundaries.	Positive 25%	Boundary 15%	Negative 20%	Insufficient 15%	Grounding 10%	FN challenge 7%	FP challenge 8%
Safety / moderation / detection Benign lookalikes vs. dangerous evasions; FP and FN challenges carry the most weight.	Positive 20%	Boundary 10%	Negative 15%	Insufficient 10%	Grounding 5%	FN challenge 20%	FP challenge 20%
Closed-book knowledge Preserve answerability; FP/FN only when a binary decision exists.	Positive 60%	Boundary 15%	Negative 10%	Insufficient 10%	Grounding 5%	FN challenge 0%	FP challenge 0%

Gaps and compliance

For a suite of N items, the target count for a type is round(target% × N). A type is on target when its share is within the declared tolerance (default ±5 percentage points) and, for any non-zero target, at least one case exists. Types with a zero target are neither required nor limited. Two remediation plans are reported:

Add-only: the smallest number of new cases that brings every type up to its target share once the denominator grows.
Rebalance at current size: add to under-filled types and revise or retire over-filled ones—only after checking topic coverage.

Composition match = 100 − ½ Σ |actual% − target%|. It summarizes distance from the profile; it is not a quality score, a pass rate or a confidence level.

Type × topic

A full bucket can still miss a critical topic. Cross case type with topic and mark applicable empty cells as gaps; explain exclusions. Do not interpret suite composition as deployment prevalence.

Human review

Coverage says what kinds of cases exist; it does not say they are correct. Review the questions and their model outputs separately under the Review Confidence Standard.