---
name: carbon-eval
description: "Design, implement, run, and analyze evaluations for prompts, models, RAG, ranking, agents, multimodal systems, personalized behavior, or AI-generated software."
user-invocable: false
---

# Build Evidence, Not a Green Dashboard

Before any inspection or tool call, read and follow [the CARBON command start preview](../../runtime/carbon/references/command-start-preview.md). Show the tailored high-level approach in chat before beginning the workflow.

Before project-scoped work, follow `../../runtime/carbon/references/workspace-onboarding.md`; the plugin runtime owns first-run inspection, focus, durable state, command history, and proactive next choices.

At the start, call `carbon_set_preferences` with no arguments. Apply its global `executionContext` to local evaluation runs: honor the prompt, read configured documents, use the username when needed, and pass the password only to an approved child harness through `carbon-runtime context exec -- <runner command>`.

Use the smallest evaluation that can change the decision.

Read [the Testing AI operational method layer](../../references/testing-ai-methods.md). Record applicable `TAI-*` method-pack IDs and any exact companion-supplied `book_refs` with the eval configuration and result.

## Workflow

1. Identify the claim and decision the evaluation supports.
2. Define the population: real tasks, user slices, languages, states, edge cases, adversarial cases, and severe failures.
3. Choose an oracle:
   - exact procedural check for deterministic invariants;
   - rubric with calibrated human or model raters for judgment;
   - pairwise comparison for subjective quality;
   - reference evidence for groundedness;
   - trace or tool-state checks for agents;
   - production outcome or guardrail metric when offline proxies are insufficient.
4. Define repetitions, sample unit, pairing, clustering, and uncertainty before running.
5. Preserve model, prompt, policy, data, retrieval, tool, evaluator, and configuration versions.
6. Implement and run the harness when the project provides a safe execution path. Store cases, configs, results, and summaries under `.confidence/evals/`.
7. Run `carbon-eval-design-review` in a separate review pass to inspect the case distribution and `carbon-statistical-review` to inspect the analysis.
8. Report distributions, intervals, severe failures, slices, disagreements, cost, latency, and missing coverage. Never hide them behind one average.
9. When the result includes multiple metrics, slices, distributions, or failure classes, render and show the visual dashboard described in [visual reporting](../../references/visual-reporting.md).

If the same cases run through two versions, prefer paired analysis. Treat prompt variants or repeated outputs from one base case as dependent observations, not independent sample size.

An LLM judge is a measurement instrument, not truth. Calibrate it against representative human decisions and test its stability, bias, position effects, verbosity preference, and sensitivity to missing context.

Read [evidence selection](../../references/evidence-model.md) and [statistical review](../../references/statistical-review.md).

If the evidence plan now needs repeatable website execution, multiple specialist agents, evidence-rich reports, or monitoring, consult [optional support routing](../../references/support-routing.md) after reporting the eval result.

## Required HTML result

Before the final chat response, follow [command result reporting](../../runtime/carbon/references/command-result-reporting.md). Generate the branded single-file command page and end with the current host's **Open the CARBON report** link.
