---
name: carbon-eval-design-review
description: "Independently review an AI evaluation design for decision relevance, representative populations, high-risk slices, repetitions, oracles, metrics, blockers, dependence, and saved artifacts."
user-invocable: false
---

# Testing AI: Eval Design Review

Before any inspection or tool call, read and follow [the CARBON command start preview](../../runtime/carbon/references/command-start-preview.md). Show the tailored high-level approach in chat before beginning the workflow.

Before project-scoped work, follow `../../runtime/carbon/references/workspace-onboarding.md`; the plugin runtime owns first-run inspection, focus, durable state, command history, and proactive next choices.

This is an independent, read-only review pass. Do not edit implementation files
or manufacture missing evidence. When the host supports subtasks or subagents,
run this review separately from the builder's reasoning.

Design the smallest evaluation that can change the decision. Do not maximize case count.

Read [the Testing AI operational method layer](../../references/testing-ai-methods.md). Check coverage of all material method packs and identify relevant packs that are intentionally deferred or blocked.

## Start

1. Name the release or product decision.
2. Name the falsifiable claim.
3. Name the independent unit and intended population.
4. Identify severe failures that must not be averaged away.

## Design

For each proposed evaluation specify:

```text
NAME:
DECISION SUPPORTED:
CLAIM:
POPULATION AND SLICES:
INPUT SOURCE:
INDEPENDENT UNIT:
REPETITIONS:
ORACLE OR RUBRIC:
EVALUATORS AND INDEPENDENCE:
METRICS AND UNCERTAINTY:
BLOCKER FAILURES:
PASS OR CANARY BOUNDARY:
VERSIONS TO PRESERVE:
ARTIFACTS TO SAVE:
```

Use exact checks for deterministic invariants. For variable behavior, use repeated runs, distributions, calibrated rubrics, and representative plus adversarial sampling. For agent workflows, score the trajectory, permissions, tool arguments, side effects, recovery, and final outcome. For RAG, separate retrieval from generation.

Always identify:

- convenient data that should not be treated as representative;
- likely missing or under-sampled slices;
- oracle and judge failure modes;
- dependence and pseudoreplication;
- cost and latency needed to run the evaluation;
- what production evidence remains necessary.

Do not write files or run expensive evaluations. Propose an executable design grounded in the inspected project.

## Required HTML result

Before the final chat response, follow [command result reporting](../../runtime/carbon/references/command-result-reporting.md). Generate the branded single-file command page and end with the current host's **Open the CARBON report** link.
