# Statistical Review

Statistics do not rescue a bad sample, weak oracle, or undefined decision.

## Start With the Unit

Name the independent unit: user, conversation, query, task, repository, session, device, or incident. Repeated outputs from one base prompt are useful for variance measurement but do not automatically create independent sample size.

## Comparison Choice

- Same cases through A and B: analyze per-case paired differences.
- Different randomly assigned units: use an independent comparison.
- Ordinal rubric scores: consider bootstrap or permutation methods and report the score distribution.
- Clustered prompts, users, or sessions: preserve clusters in resampling or modeling.
- Small or irregular samples: prefer robust nonparametric or permutation approaches over unsupported normality assumptions.

## Always Inspect

- effect size and practical importance;
- confidence or uncertainty interval;
- outliers and severe failures;
- distribution shape;
- multiple comparisons and repeated peeking;
- missing data and exclusions;
- sample-selection mechanism;
- dependence and pseudoreplication;
- whether the analysis was specified before seeing results.

Overlapping marginal 95% confidence intervals are not a formal test of the difference. When the same eval cases run through both versions, prefer an interval on the paired per-case differences.

## Calibration

When the system emits confidence, test whether confidence matches observed frequency. Use reliability plots and proper scoring rules such as Brier score when appropriate. Evaluate threshold selection, abstention, and selective prediction by risk slice. A confident system that is wrong on rare high-impact cases is not calibrated for the decision.

P-values are continuous evidence under assumptions, not permission to ship. Interpret them with the test plan, effect size, uncertainty, multiplicity, data quality, and prior expectations.
