CARBON — Engineering ConfidenceUser Guide
CHAPTER 44 / 60 Pro

/carbon-statistical-review

Review AI evaluation statistics for paired versus independent design, ordinal scores, clustered prompts, repeated runs, calibration, confidence intervals, multiple comparisons, pseudoreplication, and practical significance.

Jay · Your AI test managerAI evaluations

When to use it

You want to know whether an apparent evaluation improvement is credible.

Before you start

Preserve paired outcomes and repeated-run structure. Check clustering, sample balance, uncertainty, multiple comparisons and practical effect size; a higher average alone is not a reliable win.

Try this in your coding agent.

/carbon-statistical-review Review whether the improvement in evals/results is statistically meaningful.

Replace sample paths with your project. Slash names are shorthand; your host may use a plugin prefix or a skill picker. Invocation help →

ILLUSTRATIVE WALKTHROUGH

Evaluation comparison

An expanded teaching example, not a recorded or live execution. The report stays visible while the replay explains how to read it.

  1. 01Read context
  2. 02Analyze
  3. 03Show evidence
  4. 04Report

Coding-agent conversation

Read context

I’m reviewing evaluation comparison. I’ll keep the scope and permissions explicit.

Analyze

Analyze: Review whether the improvement in evals/results is statistically meaningful.

Show evidence

Here is the evidence trail to inspect: Check paired outcomes, uncertainty, sample balance and multiple comparisons.

Report

The apparent improvement is inconclusive under the available sample. No numeric significance claim is invented for this demo.

CARBON

Example report

Illustration
Answer vs supporting source
Answer vs supporting source — TestBucks teaching screen

TestBucks course screen with illustrative analysis. The screenshot supplies context; the demo does not establish a newly observed bug.

  1. Question / intent
  2. Reference evidence
  3. Scoring & uncertainty

Illustrated outcome

The apparent improvement is inconclusive under the available sample.

Evidence to inspect

Check paired outcomes, uncertainty, sample balance and multiple comparisons.

Still untested / not established

No numeric significance claim is invented for this demo.

Ready. Read the full example or play its steps.

What a useful result contains

A bounded claim

For evaluation comparison, state the target, revision, environment and actual work performed. Keep planned, executed, blocked and untested work separate.

An inspectable trail

Check paired outcomes, uncertainty, sample balance and multiple comparisons. In a real run, retain the source, trace, screenshot or fixture behind the conclusion.

The qualification

No numeric significance claim is invented for this demo.

The next decision

Decide whether the evidence calls for a focused check, a reviewed fix or more context. Do not convert a plausible example into a verified finding.

Adapt the example to your project

  1. Replace evaluation comparison with a real feature, path or decision in your project.
  2. State the expected rule and its source. Include safe fixtures, constraints and a budget where appropriate.
  3. Ask for exact evidence and remaining gaps. Approve consequential actions separately.
/jay Review the /carbon-statistical-review result. Separate observed evidence, assumptions and untested scope, then recommend the next useful check.

Where to go next

/carbon-eval-design-review →/carbon-release-review →

Content checked against the local command definition: skills/carbon-statistical-review/SKILL.md. Examples adapted from the homepage’s TestBucks walkthroughs. Availability is edition- and host-dependent.