# CARBON chatbot testing reference

Use this reference to turn product context into a risk-traceable conversational suite. Breadth is required, but every case must have a product reason and a reviewable oracle.

## Context model

Capture the chatbot's goals, supported and unsupported intents, users and personas, high-consequence failures, source-of-truth documents, tools and permissions, memory model, handoff behavior, supported channels, locales, data classifications, latency and availability targets, and release blockers.

Extract truth in this order: user-confirmed requirements, authoritative product artifacts, code and schemas, observed behavior, then labeled inference. Conflicts become explicit open questions and tests; they are not silently resolved.

## Coverage families

1. **Core task success** — greetings, capability discovery, supported intents, goal completion, partial completion, unsupported requests, human handoff, and graceful closure.
2. **Instruction following** — explicit constraints, ordered requests, negative constraints, output formats, length limits, tone, role boundaries, and conflicting instructions.
3. **Ambiguity and clarification** — underspecified requests, multiple plausible intents, corrections, misspellings, contradictory details, missing fields, and premature action prevention.
4. **Accuracy and grounding** — product facts, prices, policies, calculations, dates, citations, source attribution, source freshness, conflicting sources, unknown facts, and calibrated uncertainty.
5. **Hallucination** — fabricated facts, capabilities, actions, transactions, citations, people, URLs, confirmation numbers, or success claims; unsupported assumptions; false memory.
6. **Context and memory** — pronouns, references, earlier constraints, correction, topic switching, long conversations, context-window pressure, new-session isolation, deletion/forget requests, and contradiction recovery.
7. **Safety and fallback** — harmful, illegal, self-harm, medical, legal, financial, age-sensitive, abusive, and emergency content; proportional refusal; useful safe redirection.
8. **Security and adversarial behavior** — direct and indirect prompt injection, jailbreaks, encoding/obfuscation, role-play attacks, system-prompt extraction, delimiter attacks, tool-output injection, RAG poisoning, and instruction hierarchy.
9. **Privacy and data boundaries** — PII requests, minimization, secrets, credentials, payment data, retention claims, cross-user and cross-tenant leakage, private source disclosure, data deletion, and consent.
10. **Tool and action use** — selection, parameters, authorization, confirmation, least privilege, side effects, idempotency, retries, partial failure, stale results, cancellation, and honest reporting of action status.
11. **Retrieval and RAG** — relevant retrieval, citation faithfulness, no-result behavior, access controls, conflicting passages, stale indexes, malicious documents, chunk boundaries, and answer/source consistency.
12. **Personas, bias, and fairness** — role and expertise differences, protected characteristics, dialects, socioeconomic contexts, assistive needs, equitable service, stereotype resistance, and consistent policy.
13. **Robustness and input boundaries** — empty and whitespace input, one character, very long and effectively unbounded input, powers-of-two boundaries, emoji, emoticons, Unicode, combining marks, RTL, markup, code, URLs, attachments, repeated messages, typos, slang, and hostile formatting.
14. **Localization** — supported and unsupported languages, mixed language, locale-specific date/number/currency, names and addresses, translation fidelity, RTL presentation, and locale persistence.
15. **Conversation UX** — clarity, brevity, empathy, brand voice, scannability, progressive disclosure, recovery prompts, edit/retry/regenerate, feedback controls, close/exit, and truthful status.
16. **Accessibility and interface integration** — keyboard use, focus, labels, live-region announcements, contrast, zoom, reduced motion, screen reader semantics, streaming updates, launcher state, consent gates, and mobile layout.
17. **Performance and resilience** — first-token and full-response latency, streaming stalls, timeout, offline behavior, retry/backoff, rate limits, concurrency, load, queueing, availability, and resource ceilings.
18. **API and protocol behavior** — authentication, authorization, content types, schemas, validation errors, status codes, streaming/SSE termination, webhooks, ordering, idempotency keys, pagination, and trace correlation.
19. **Non-determinism and regression** — repetitions, paraphrases, metamorphic invariants, temperature/model/prompt changes, drift, flaky behavior, stability bands, prior failures, and release comparisons.
20. **Domain-specific risk** — product vocabulary, regulatory duties, business rules, escalation paths, money movement, account state, irreversible actions, and known incident patterns.

## Case design patterns

- **Positive:** realistic supported request with valid state and data.
- **Negative:** unsupported, invalid, unauthorized, or unsafe request.
- **Boundary:** just below/at/above documented and implementation limits; include 0, 1, max-1, max, max+1 and powers of two where transports or token limits matter.
- **Multi-turn:** establish facts or constraints, switch context, correct one fact, then verify relevant memory without unrelated leakage.
- **Metamorphic:** paraphrase, reorder harmless details, or change irrelevant attributes; the decision-relevant answer should remain equivalent.
- **Persona:** vary expertise, role, locale, accessibility need, and emotional state without changing entitlement.
- **Adversarial:** place untrusted instructions in user text, retrieved content, tool output, files, URLs, or quoted prior messages.
- **Recovery:** induce timeout, failed tool, stale data, interruption, or contradiction; verify honest status and useful recovery.
- **Repeated:** run high-risk semantic cases at least three times; use more repetitions when consistency is itself the claim.

## Oracle model

Use one or more explicit oracle types:

- `deterministic`: schema, exact fact, permitted tool, status code, required field, or authoritative state.
- `semantic`: meaning-based criteria with allowed variation and prohibited claims.
- `invariant`: properties that must hold across paraphrases, repetitions, personas, or sessions.
- `human-review`: domain judgment, tone, fairness, nuanced safety, or unresolved product expectation.

Each oracle says what evidence supports it. A response existing is transport evidence, not semantic success. Do not call an inferred expectation validated.

## Suite payload

Pass this shape to `carbon-runtime chatbot render`:

```json
{
  "schema": "carbon.chatbot-suite-input/v1",
  "suite": {
    "id": "support-assistant",
    "name": "Support assistant",
    "description": "Risk-based conversational validation",
    "target": {"kind": "website", "url": "https://example.test/chat", "entryHint": "Open Help"},
    "contextSummary": "Redacted summary of supplied and inspected context",
    "goals": ["Resolve supported requests accurately"],
    "risks": ["Invented policy", "cross-user leakage"],
    "personas": [{"id": "new-customer", "name": "New customer", "description": "First-time user"}],
    "sources": [{"kind": "requirement", "label": "Support policy", "path": "docs/support.md", "evidence": "observed"}],
    "openQuestions": ["What is the approved escalation path?"],
    "tests": [
      {
        "id": "CHAT-ACC-001",
        "title": "Ground refund answer in current policy",
        "category": "Accuracy and grounding",
        "type": "multi-turn",
        "priority": "P0",
        "risk": "Invented refund policy causes customer harm",
        "rationale": "Refund policy is a high-consequence supported intent",
        "provenance": [{"kind": "source", "ref": "docs/refunds.md"}],
        "persona": "new-customer",
        "preconditions": ["Fresh isolated conversation"],
        "setup": ["Use synthetic order data"],
        "data": {"orderAgeDays": 45},
        "turns": [{"role": "user", "content": "Can I return an opened item after 45 days?"}],
        "expectedBehavior": ["States the documented policy or clearly says it cannot verify"],
        "prohibitedBehavior": ["Invents an exception or claims a refund was issued"],
        "oracles": [{"type": "semantic", "criterion": "Answer matches docs/refunds.md", "evidence": "reply and source snapshot"}],
        "evidenceRequirements": ["complete transcript", "source/build identity", "latency"],
        "repetitions": 3,
        "tags": ["policy", "high-consequence"],
        "dependencies": [],
        "safety": {"destructive": false, "paid": false, "public": false, "highVolume": false},
        "status": "planned"
      }
    ]
  }
}
```

The renderer accepts `prompt` or `input` as a shorthand for one user turn and `expected` or `passCriteria` as shorthand for expected behavior, including imports from `icebergqa.chatbot-project/v1`. New CARBON cases should use the richer fields.

## Existing suite behavior

`extend` merges by stable ID and a normalized fingerprint of category, title, and turns. Exact and semantic-looking duplicates are omitted and reported. Previous JSON is saved under `versions/` before changes. `new` creates a timestamped suite ID when the base ID exists. History is never destroyed.

## Execution evidence

Capture target/build/model/prompt identity where available, isolated-session ID, full transcript, timestamps, latency, tool calls, source citations, screenshots or network evidence when appropriate, oracle decisions, human-review notes, and explicit `passed`, `failed`, `blocked`, `skipped`, or `not-run` status. Preserve partial results on interruption.
