# Evidence Selection

Match the evidence to the claim and failure mode.

| Claim or risk | Useful evidence |
| --- | --- |
| Deterministic invariant | Unit, integration, property, schema, type, static, or procedural check |
| Variable answer quality | Representative cases, repeated runs, calibrated rubric, distributions, intervals |
| Regression between versions | Same cases through both versions, paired differences, severe-failure comparison |
| Retrieval quality | Context recall and precision, source freshness, ranking, citation faithfulness |
| Tool-using agent | Trajectory traces, permission checks, side-effect assertions, recovery and approval tests |
| Bias or uneven value | Defined slices, counterfactual cases, representative raters, per-slice uncertainty |
| Safety or security | Threat model, adversarial cases, untrusted-channel tests, containment and denial traces |
| Production readiness | Shadow or canary evidence, latency and cost distributions, SLOs, rollback drill |
| Subjective preference | Pairwise comparison, rater calibration, disagreement analysis |

## The Evidence Ladder

Weak to stronger:

1. One demo or one passing run.
2. Builder-authored checks.
3. Repeated tests on a convenient set.
4. Representative and adversarial samples with preserved versions.
5. Independent procedural, model, tool, or human evaluation.
6. Production evidence with guardrails, monitoring, and tested rollback.

Higher does not always mean better. Use the lowest-cost evidence that can detect the decision-relevant failure. High-risk and irreversible actions justify a stronger ladder position.

## Required Questions

- What exact claim does this evidence support?
- What population does the sample represent?
- What severe failures can the metric hide?
- What dependence or shared blind spot exists between builder and evaluator?
- What would falsify the claim?
- What evidence is missing?
