# Testing AI Operational Method Layer

CARBON includes the operational methods needed to turn AI-quality concerns into runnable tests and release evidence. It does not bundle or duplicate the complete *Testing AI* book. The separate Testing AI plugin remains the chapter-level reasoning and reference product.

## Three-Layer Architecture

1. **Testing AI book plugin — explain and select.** It can inspect a project against all book chapters and themes, cite exact local references, and produce book-guided findings.
2. **CARBON method layer — operationalize.** CARBON converts the applicable principles into risks, falsifiable claims, eval cases, browser or API tests, statistics, monitoring, and release gates.
3. **CARBON execution layer — observe.** Local CARBON and explicitly selected testers.ai Premium Cloud workflows run the checks and preserve evidence. Planned or generated work remains unverified until execution supports it.

Neither plugin is a hard dependency of the other. CARBON must remain useful when the book plugin is absent. The book plugin must remain useful when CARBON is absent.

## Operational Method Packs

Apply every pack that is material to the inspected system; do not force every pack into every task.

| ID | Method pack | CARBON application | Book lineage |
|---|---|---|---|
| `TAI-UNCERTAINTY` | Non-determinism, sampling, repeated runs, distributions, uncertainty, rare failures | `carbon-confidence-plan`, `carbon-eval`, `carbon-statistical-review` | End of One-Run Testing; Sampling and Uncertainty; anti-patterns about demos, pass rates, and aggregates |
| `TAI-ORACLES` | Exact invariants, metamorphic properties, rubrics, pairwise comparison, judges, human calibration, disagreement | `carbon-eval`, `carbon-eval-design-review`, `carbon-skeptical-review` | Tests to Release Evidence; Judges, Humans, and Disagreement; Building Evals That Matter |
| `TAI-STATISTICS` | Independent unit, pairing, clustering, intervals, effect size, power, multiplicity, practical significance | `carbon-statistical-review`, `carbon-release-review` | Statistical Tests for AI Quality |
| `TAI-SYSTEMS` | RAG, agents, tool trajectories, data contracts, traces, cost, latency, versioning | `carbon-confidence-plan`, `carbon-eval`, `carbon-security-review` | Release Readiness; Operating AI; AI Security and Guardrails |
| `TAI-GENERATED-SOFTWARE` | AI-generated code, integration mistakes, security/privacy, maintainability, generated-test coverage illusions | `carbon-review`, `ai-upgrade`, `carbon-browser` | Generated Code Changes the Job; Coding-agent review loops |
| `TAI-BIAS` | Population coverage, slices, counterfactuals, raters, cultural/language/accessibility bias, feedback loops | `carbon-confidence-plan`, `carbon-eval-design-review`, `carbon-release-review` | Data, Bias, Raters, and Incentives |
| `TAI-SECURITY-SAFETY` | Threat models, prompt injection, poisoning, tool permissioning, containment, manipulation, deception, hazardous capability boundaries | `carbon-security-review`, `carbon-incident`, `carbon-release` | AI Security and Guardrails; Frontier Safety and Containment |
| `TAI-OPERATIONS` | Monitoring, canary, shadow, rollback, production trace mining, incident promotion, EvalOps | `carbon-release`, `carbon-incident`, `carbon-auto` | Release Readiness; Operating AI; Practical Playbook |
| `TAI-SPECIALIZED` | Multimodal, model internals, personalization, embodied/long-running systems, governance, regulation | Route to the relevant CARBON evaluator and specialist reviews; narrow scope and state tool limits | Model behavior and introspection; dynamic AI products; embodied systems; governance and moral futures |

These IDs are stable CARBON provenance labels, not substitutes for chapter citations. When the Testing AI plugin is used, preserve its exact book references alongside the method-pack IDs.

## Required Rules

- Start from a decision and a falsifiable claim, not a test count.
- Separate deterministic invariants from variable behavior.
- Treat one run, one average, and one judge as weak evidence.
- Define the intended population, high-risk slices, independent unit, repetitions, and severe failures before interpreting results.
- Preserve model, prompt, policy, data, retrieval, tool, evaluator, code, and environment versions.
- Score agent trajectories, permissions, side effects, recovery, and final outcomes—not only the final answer.
- Calibrate rubrics, judges, and human raters; preserve disagreement instead of averaging it away.
- Distinguish statistical significance from practical impact.
- Keep builder-generated tests as useful but correlated evidence; use independent review for consequential decisions.
- Promote production failures, incidents, and reviewer disagreements into durable regression cases.
- End with the evidence-supported scope, uncertainty, monitoring, rollback, and what would change the decision.

## Companion Handoff Contract

When the user asks to apply the full book, a named chapter, or whole-book reasoning and the Testing AI plugin is installed:

- Claude Code: `/testing-ai:testing-ai assess|review|plan <scope>`
- Codex: `$testing-ai assess|review|plan <scope>`
- Claude Desktop: `Use Testing AI to assess|review|plan <scope>`
- Standalone MCP: prompt `testing_ai`

The book-guided output should provide, for every material recommendation:

```text
OBSERVATION:
RISK OR UNSUPPORTED CLAIM:
METHOD_PACKS: TAI-...
BOOK_REFERENCES: exact local chapter or theme references
RECOMMENDED CHANGE:
VALIDATION:
WHAT WOULD CHANGE THE DECISION:
RESIDUAL UNCERTAINTY:
```

CARBON then maps the handoff into:

- `.confidence/risks.yaml`: `method_refs` and `book_refs` on the affected risk;
- `.confidence/claims.yaml`: the falsifiable claim, required evidence, `method_refs`, and `book_refs`;
- `.confidence/evals/`: executable cases, repetitions, slices, oracle, versions, and provenance;
- `.carbon/runs/`: runnable local product tests and observed results;
- `.confidence/release-report.md`: the resulting decision and the evidence that would change it.

Do not silently claim the companion plugin ran. Do not manufacture chapter citations when its references were not loaded. CARBON can apply the method packs directly and label exact book references as unavailable.
