# Common AI baseline

**Test family:** `AIC`

AI output is a distribution, not a single expected string. Test decision-relevant claims over representative populations and repeated runs. Preserve model/version, prompt/template, parameters, context, tools, retrieval results, and trace.

- **AIC-001** Define the user decision and measurable success/failure before choosing prompts.
- **AIC-002** Include ordinary, boundary, ambiguous, underspecified, contradictory, multilingual, adversarial, and high-impact samples.
- **AIC-003** Repeat nondeterministic cases enough to expose instability and tail failures; do not average away severe failures.
- **AIC-004** Separate model behavior from retrieval, prompt, tool, parser, UI, policy, and downstream action failures.
- **AIC-005** Check factuality/grounding, completeness, relevance, instruction fit, calibration, refusal, safety, privacy, and latency/cost.
- **AIC-006** Preserve exact inputs/outputs/traces and evaluator version; use independent or deterministic oracles for critical claims.
- **AIC-007** Compare against a baseline and important slices, not only a global average.
- **AIC-008** Empty/error/timeout/rate-limit/context-limit/tool-failure states are honest, retryable, and do not fabricate success.
- **AIC-009** Untrusted retrieved/user/tool content cannot override system policy or exfiltrate secrets.
- **AIC-010** Consequential action requires authorization, preview/confirmation where appropriate, bounded tools, idempotency, and audit.
- **AIC-011** Feedback and user data respect consent, tenant boundaries, retention, and training policy.
- **AIC-012** UI distinguishes generated/inferred content, sources, uncertainty, and last-updated/model state where relevant.

## Comprehensive per-item coverage

- **AIC-013** Test empty, minimal, ordinary, boundary-length, malformed, duplicated, and contradictory inputs; distinguish invalid input from a valid low-confidence result.
- **AIC-014** Include ASCII emoticons, emoji/ZWJ sequences, kaomoji, accents, mixed scripts, RTL, code, tables, and structured data; preserve meaning through tokenization and rendering.
- **AIC-015** Use representative easy, ambiguous, adversarial, rare, multilingual, and high-impact examples with independent expected outcomes.
- **AIC-016** Repeat identical cases across runs, seeds, model versions, and temperatures; measure outcome distribution rather than treating one run as truth.
- **AIC-017** Compare exactness, completeness, grounding, calibration, citation, abstention, and harmful-error severity with task-specific evaluators.
- **AIC-018** Inject instructions through user text, retrieved content, files, metadata, tool output, and prior messages; keep untrusted content subordinate to policy.
- **AIC-019** Request secrets, hidden prompts, other users’ data, cross-tenant context, and restricted source content; refuse without confirming existence.
- **AIC-020** Use conflicting system, developer, user, retrieved, and tool instructions; preserve the documented authority order.
- **AIC-021** Test unavailable, stale, poisoned, duplicated, and permission-restricted retrieval sources; cite only accessible evidence and expose uncertainty.
- **AIC-022** Exercise unsupported tool calls, excessive permissions, malformed arguments, slow tools, partial tool results, and tool failures; contain side effects.
- **AIC-023** Cancel, retry, regenerate, edit the prompt, branch the conversation, and resume after reconnect; keep identity, trace, and side effects coherent.
- **AIC-024** Force timeout, rate limit, context overflow, model outage, moderation failure, and dependency failure; provide a truthful fallback without fabricated completion.
- **AIC-025** Measure first-token latency, total latency, throughput, token/cost use, retries, and tail behavior for short and boundary-size inputs.
- **AIC-026** Verify streaming does not expose later-redacted content, broken encoding, partial structured values, stale citations, or inaccessible actions.
- **AIC-027** Render Markdown, code, links, tables, bidi text, emoji, and hostile markup safely; keep model output inert unless a scoped action is confirmed.
- **AIC-028** Check accessibility of prompt, progress, stop, regenerate, citations, errors, and streamed output with keyboard and screen readers.
- **AIC-029** Verify retention, deletion, training use, logging, trace access, redaction, export, and regional handling match the stated privacy contract.
- **AIC-030** Segment quality and harm metrics by language, locale, demographic proxies, input complexity, user role, and consequential use case.
- **AIC-031** Detect model, prompt, retrieval, policy, and data drift with versioned evaluations, canaries, severe-failure review, and rollback thresholds.
- **AIC-032** Require a scoped release decision based on representative repeated evidence; do not hide severe failures behind average scores.

[Back to AI intelligence behaviors index](../../ai-intelligence.md)
