EvalCARBON Eval · Eval² · Software Eval · Human Review
Point CARBON at a folder. It finds every project, every eval set and every test suite, classifies each question and test against open coverage standards, and shows you—visually—exactly what to add.
Live demos · tap in and play
These are the actual interactive reports the plugin generates, running on synthetic demo projects. Tap a project, switch the target profile, pinch or zoom the case landscape, tap any tile or matrix cell—and set your name to rate anything 👍 or 👎. Load the sample team ratings to see three personas disagree.
Support assistant, policy RAG, content moderation, ops agent and coding evals—each against its own profile.
Open the Eval² demo →
Jest + Playwright storefront, pytest billing API, Go auth service and a Vitest design system.
Open the Software Eval demo →
240 eval questions and 116 test cases, double-rated, with disagreements, adjudication—and one stratum nobody reviewed.
Open the Human Review demo →Three products · one standards family
A green build or a high eval score only means something if the suite asks the right mix of questions. Each product measures a different part of that.
For AI eval sets, benchmarks and golden datasets. Say “Eval squared.”
For unit, API, UI, end-to-end and manual test suites.
For reviewing questions, tests and their results with people.
Human Review · built into every report
Every CARBON report has rating built in. Pick who you are—name, role, photo—then tap 👍 or 👎 on any eval question or test. The whole item lights up green or red, and a comment bubble opens for the why. Switch personas to rate as different people, then compare agreement, κ and every split decision.
Add yourself and teammates—or role-play a domain expert, a skeptic and an end user—and rate as each.
Ratings and comments save in your browser and come back on reload. Nothing is uploaded.
Export a ratings file per person, merge them (or ask your coding agent to), and import the combined file back—or feed it to the next scan as review evidence.
How it works
CARBON finds every project inside—monorepos, services, notebooks—and which ones have eval sets, test suites, or neither. Read-only; nothing runs.
Each question and test gets exactly one primary case type. Your coding agent reviews the uncertain ones and records why.
Actual mix versus a declared target profile—overall and by topic or feature—with the composition match and every gap counted.
A minimal add-only plan, concrete proposed cases, and a human-review sample plan. Click, zoom and drill into any case.
The standard, in one picture
Supported · false premise · boundary · insufficient info · grounding · FP and FN challenges. All AI profiles
Positive · negative · boundary · errors & recovery · state · security · accessibility · performance. All software profiles
Every item gets one primary type, so a suite’s mix always totals 100%. Targets are proposed starting points you set per application—not universal quotas. A profile match is coverage evidence, not proof of quality.
How Human Review works
Eval questions, AI eval results, software test cases and test execution results are reviewed and reported separately. A valid question doesn’t make an answer correct, and a correct answer doesn’t rescue a broken question.
What you report
at 95% confidence · 185 of 1,240 eval questions reviewed · illustrative
Review Confidence score: breadth, precision, sampling integrity and protocol. It measures evidence sufficiency—not a pass rate. Read the standard
Open standards · OpenTest.AI
The Coverage Confidence and Review Confidence standards are proposed for industry discussion and will be maintained at OpenTest.AI—open testing for the AI era. CARBON is the first product built on them. These drafts are hosted here until they move.
Runs where you build
The scan is local and read-only: no uploads, no project code executed. Your coding agent does the judgment calls—relabeling uncertain cases and writing concrete proposals—then opens the report.
Early access is by request while we pilot with teams.
Pick a product. We’ll set you up and walk through your first report together.