---
name: carbon-statistical-review
description: "Review AI evaluation statistics for paired versus independent design, ordinal scores, clustered prompts, repeated runs, calibration, confidence intervals, multiple comparisons, pseudoreplication, and practical significance."
user-invocable: false
---

# Testing AI: Statistical Review

Before any inspection or tool call, read and follow [the CARBON command start preview](../../runtime/carbon/references/command-start-preview.md). Show the tailored high-level approach in chat before beginning the workflow.

Before project-scoped work, follow `../../runtime/carbon/references/workspace-onboarding.md`; the plugin runtime owns first-run inspection, focus, durable state, command history, and proactive next choices.

This is an independent, read-only review pass. Do not edit implementation files
or manufacture missing evidence. When the host supports subtasks or subagents,
run this review separately from the builder's reasoning.

You are reviewing a measurement argument, not decorating a report with statistics.

Read [the Testing AI operational method layer](../../references/testing-ai-methods.md). Apply `TAI-STATISTICS`, and preserve the linked claim, population, and any companion-supplied book references.

## Determine

- the independent unit;
- whether A and B observations are paired;
- whether rubric scores are ordinal or meaningfully numeric;
- whether repeated runs, prompts, users, or sessions are clustered;
- whether exclusions and missing data alter the sample;
- whether the sampling frame represents the release population;
- whether multiple comparisons or repeated peeking inflate false discoveries;
- whether effect size matters to users or the business;
- whether intervals and uncertainty match the design;
- whether confidence or abstention is calibrated by risk slice.

## Prefer

- paired per-case differences when the same cases run through A and B;
- bootstrap or permutation approaches for irregular, ordinal, or small samples;
- cluster-aware resampling or models when observations share a base prompt, user, repository, or session;
- distributions and severe-failure rates alongside averages;
- predeclared metrics and thresholds;
- practical significance with cost, latency, and safety constraints.

Do not use overlap between two marginal 95% intervals as a formal test of the difference. Do not treat repeated generations from one prompt as independent prompts. Do not imply that a low p-value proves product wisdom.

## Output

```text
DESIGN AS UNDERSTOOD:
  independent unit:
  pairing:
  clustering:
  outcome type:

VALID ANALYSES:
  - ...

THREATS TO VALIDITY:
  - ...

MISSING REPORTING:
  - ...

PRACTICAL INTERPRETATION:
  <what the numbers can and cannot support>

STATISTICAL POSITION: adequate | adequate-with-conditions | invalid | insufficient-information
```

Do not edit data, code, or reports.

## Required HTML result

Before the final chat response, follow [command result reporting](../../runtime/carbon/references/command-result-reporting.md). Generate the branded single-file command page and end with the current host's **Open the CARBON report** link.
