CARBON — Engineering ConfidenceUser Guide
CHAPTER 42 / 60 Pro

/carbon-eval

Design, implement, run, and analyze evaluations for prompts, models, RAG, ranking, agents, multimodal systems, personalized behavior, or AI-generated software.

Diego · AI chatbotAI evaluations

When to use it

You need evidence about model, prompt, retrieval or agent behavior.

Before you start

Define intended use, representative cases, harmful failures and an explicit rubric. Retain inputs, outputs, source evidence and scoring rationale; a judge score without an oracle is not enough.

Try this in your coding agent.

/carbon-eval Design and run local evaluations for the search assistant using synthetic questions.

Replace sample paths with your project. Slash names are shorthand; your host may use a plugin prefix or a skill picker. Invocation help →

ILLUSTRATIVE WALKTHROUGH

Search-assistant evaluation

An expanded teaching example, not a recorded or live execution. The report stays visible while the replay explains how to read it.

  1. 01Read context
  2. 02Investigate
  3. 03Show evidence
  4. 04Report

Coding-agent conversation

Read context

I’m reviewing search-assistant evaluation. I’ll keep the scope and permissions explicit.

Investigate

Follow the weakest evidence link and record what remains blocked.

Show evidence

Here is the evidence trail to inspect: Synthetic question → retrieved passage → answer → groundedness rubric → failure explanation.

Report

A correct-looking answer cites a source that does not support its claim. Rubric and sample coverage need review; one case is not a benchmark.

CARBON

Example report

Illustration
Answer vs supporting source
Answer vs supporting source — TestBucks teaching screen

TestBucks course screen with illustrative analysis. The screenshot supplies context; the demo does not establish a newly observed bug.

  1. Question / intent
  2. Reference evidence
  3. Scoring & uncertainty

Illustrated outcome

A correct-looking answer cites a source that does not support its claim.

Evidence to inspect

Synthetic question → retrieved passage → answer → groundedness rubric → failure explanation.

Still untested / not established

Rubric and sample coverage need review; one case is not a benchmark.

Ready. Read the full example or play its steps.

What a useful result contains

A bounded claim

For search-assistant evaluation, state the target, revision, environment and actual work performed. Keep planned, executed, blocked and untested work separate.

An inspectable trail

Synthetic question → retrieved passage → answer → groundedness rubric → failure explanation. In a real run, retain the source, trace, screenshot or fixture behind the conclusion.

The qualification

Rubric and sample coverage need review; one case is not a benchmark.

The next decision

Decide whether the evidence calls for a focused check, a reviewed fix or more context. Do not convert a plausible example into a verified finding.

Adapt the example to your project

  1. Replace search-assistant evaluation with a real feature, path or decision in your project.
  2. State the expected rule and its source. Include safe fixtures, constraints and a budget where appropriate.
  3. Ask for exact evidence and remaining gaps. Approve consequential actions separately.
/jay Review the /carbon-eval result. Separate observed evidence, assumptions and untested scope, then recommend the next useful check.

Where to go next

/carbon-eval-design-review →/carbon-statistical-review →

Content checked against the local command definition: skills/carbon-eval/SKILL.md. Examples adapted from the homepage’s TestBucks walkthroughs. Availability is edition- and host-dependent.