CARBON — Engineering ConfidenceUser Guide
CHAPTER 43 / 60 Pro

/carbon-eval-design-review

Independently review an AI evaluation design for decision relevance, representative populations, high-risk slices, repetitions, oracles, metrics, blockers, dependence, and saved artifacts.

Diego · AI chatbotAI evaluations

When to use it

The evaluation itself may be measuring the wrong thing.

Before you start

Inspect case provenance, population coverage, difficult negatives, holdout overlap and scoring. Review decision relevance before spending time on a larger run with the same design flaw.

Try this in your coding agent.

/carbon-eval-design-review Review evals/search for coverage gaps, leakage and misleading success criteria.

Replace sample paths with your project. Slash names are shorthand; your host may use a plugin prefix or a skill picker. Invocation help →

ILLUSTRATIVE WALKTHROUGH

Evaluation design

An expanded teaching example, not a recorded or live execution. The report stays visible while the replay explains how to read it.

  1. 01Read context
  2. 02Analyze
  3. 03Show evidence
  4. 04Report

Coding-agent conversation

Read context

I’m reviewing evaluation design. I’ll keep the scope and permissions explicit.

Analyze

Analyze: Review evals/search for coverage gaps, leakage and misleading success criteria.

Show evidence

Here is the evidence trail to inspect: Inspect case provenance, holdout overlap, scoring rules and boundary controls.

Report

The dataset repeats answer wording and omits difficult negatives. This reviews the evaluation design, not model performance.

CARBON

Example report

Illustration
Answer vs supporting source
Answer vs supporting source — TestBucks teaching screen

TestBucks course screen with illustrative analysis. The screenshot supplies context; the demo does not establish a newly observed bug.

  1. Question / intent
  2. Reference evidence
  3. Scoring & uncertainty

Illustrated outcome

The dataset repeats answer wording and omits difficult negatives.

Evidence to inspect

Inspect case provenance, holdout overlap, scoring rules and boundary controls.

Still untested / not established

This reviews the evaluation design, not model performance.

Ready. Read the full example or play its steps.

What a useful result contains

A bounded claim

For evaluation design, state the target, revision, environment and actual work performed. Keep planned, executed, blocked and untested work separate.

An inspectable trail

Inspect case provenance, holdout overlap, scoring rules and boundary controls. In a real run, retain the source, trace, screenshot or fixture behind the conclusion.

The qualification

This reviews the evaluation design, not model performance.

The next decision

Decide whether the evidence calls for a focused check, a reviewed fix or more context. Do not convert a plausible example into a verified finding.

Adapt the example to your project

  1. Replace evaluation design with a real feature, path or decision in your project.
  2. State the expected rule and its source. Include safe fixtures, constraints and a budget where appropriate.
  3. Ask for exact evidence and remaining gaps. Approve consequential actions separately.
/jay Review the /carbon-eval-design-review result. Separate observed evidence, assumptions and untested scope, then recommend the next useful check.

Where to go next

/carbon-eval →/carbon-statistical-review →

Content checked against the local command definition: skills/carbon-eval-design-review/SKILL.md. Examples adapted from the homepage’s TestBucks walkthroughs. Availability is edition- and host-dependent.