Review Confidence Standard
How much human review is enough to trust an eval or a test suite—and how to say so honestly. Report the estimated acceptable rate for a declared population with its interval, confidence level and sample size.
Four review populations
Eval questions
Is each question valid, answerable, correctly keyed and classified?
AI eval results
Is each model output correct and grounded against its frozen criterion?
Software test cases
Does each test check the right behavior with a sound oracle?
Test execution results
Do logs, screenshots and environment support the recorded outcome?
Store and report these separately. A valid question does not make a response correct; a correct response does not rescue a defective question.
What to report
- Declare the population, strata, sample plan and precision target before reviewing.
- Use probability sampling for population claims; hand-picked reviews stay descriptive.
- Report lower and upper bounds when intervals are asymmetric (exact finite-population intervals).
- Unreviewed groups and critical failures stay visible as unknown—they never disappear from the denominator.
- Reviewer agreement is reported separately; agreement is not correctness.
Risk tiers and sample sizes
Conservative planning size n = ⌈ln(2/α) / (2h²)⌉ for half-width h, capped at a full census.
| Tier | Confidence | Half-width | Planning sample |
|---|---|---|---|
| low | Confidence 95% | Half-width ±10 pp | Sample 185 (or census if smaller) |
| moderate | Confidence 95% | Half-width ±5 pp | Sample 738 (or census if smaller) |
| high | Confidence 99% | Half-width ±5 pp | Sample 1060 (or census if smaller) |
| critical | Confidence 99% | Half-width ±2 pp | Sample 6623 (or census if smaller) |
Review Confidence score
B reviewed breadth across required strata · P precision attained versus target half-width · S sampling integrity and traceability · A review protocol completion (independent ratings, dispositions). The score measures evidence sufficiency; it is not a pass rate or a confidence level.