AI Bug-Finder Leaderboard

AI-generated pages with deliberately seeded bugs

Every score on this page comes from a held-out cohort of three AI-generated web pages — a search-engine results page, a news article, and a social feed. Each page was produced by an LLM coding tool and then deliberately seeded with a known set of defects across a range of categories: accessibility violations, missing security headers, broken links, console errors, layout overflow, content typos, performance issues, and so on.

The seeded bugs explicitly include the failure modes AI coding tools commonly ship — missing alt text on images, hardcoded credentials in client-side code, unsanitised user input, dead-code links, broken aria-labels, mixed-protocol resources, and the long tail of "looks fine, fails when reviewed" defects that show up when an LLM writes the code. That's the point: this benchmark scores how well bug-finder agents catch the bugs that AI-generated code is actually producing right now.

Each page is fully self-contained (inline CSS, inline SVG, no network access required at scoring time). Bug counts per page are tracked in the eval’s ground_truth.json alongside each page’s artifacts.

Each model receives the same artifact bundle: rendered HTML, browser console log, network transcript (with response bodies), and a screenshot. No live browser, no network access, no model-specific harness tricks — just the four artifacts and a shared system prompt.

An independent LLM-as-judge then matches each model’s findings against the seeded ground-truth bugs. Matched findings count as true positives. Unmatched findings — potential false positives — go through a second-pass classifier that asks: "is this finding pointing at something real on the page that we just didn’t seed, or did the model fabricate this?" That second pass is what powers the Groundedness metric and lets us reward models for finding eval gaps rather than punishing them for it.

Why this matters: every metric below has a known ground-truth denominator and a calibrated FP classifier. The numbers are reproducible — same artifacts, same judge prompt, same scoring math — even though the underlying models change.

Determinism: every model is invoked with decoding parameters set as deterministic as the provider allows — temperature=0.1, top_p=0.9 (or provider equivalents; reasoning-tier models that reject these are run with their stricter defaults). Every score on this page is averaged across the three pages in the cohort, not measured on one.

Overall — equal-weighted average across all five metrics

What it measures: A single composite score combining F_0.5, Discovery F_0.5, Precision, Recall, and Groundedness. No bias toward any particular error type — just "how does this model do across the whole evaluation?"

Math: Overall = (F0.5 + Discovery + Precision + Recall + Groundedness) / 5. Each component is in [0, 1]; the average is also in [0, 1].

Best for teams that: want the single safest pick across mixed workloads — an evaluation pass with no specific cost asymmetry, comparison tables for executives, "which model should we default to?" decisions.

Note: the ★ tuned for this badge appears on the five specialized boards below to mark the testers.ai mode tuned for that metric. It doesn’t apply to Overall — there is no dedicated optimize_for=overall mode; it’s an unweighted average of the five.

F_0.5 — precision-weighted F-score (the headline metric)

What it measures: A balance of precision (the fraction of flagged bugs that are real) and recall (the fraction of real bugs that were caught), weighted so that false positives hurt 2× more than misses. In other words: a model that catches 9 of 10 bugs cleanly beats one that catches all 10 buried under a pile of false alarms. Range 0–100%, higher is better.

Math: Fβ = (1+β²) · P · R / (β²·P + R) with β=0.5 → F0.5 = 1.25 · P · R / (0.25·P + R). Compare with F1 (β=1, equal weight on P and R) and F2 (β=2, recall-weighted).

Best for teams that: triage a real bug backlog, where every false alarm eats engineer time; CI gates that fail builds on AI findings; bug-tracker auto-assignment; any workflow where a person reads each finding and a wrong one costs more than a missing one.

Skip this metric if: you're running a security audit (use Recall), or feeding output to an automated pipeline (use Groundedness).

Discovery F_0.5 — weighted reward for finding rare bugs

What it measures: Like F_0.5, but recall is rarity-weighted: a bug caught by only one of the 15 models counts 15× more than a bug everyone catches. Rewards finding the long-tail issues your other tools miss; a model that just rediscovers what every model already finds gets a low score here.

Math: Replace plain recall with RareR = Σb∈caught(1/nb) / Σb∈GT(1/nb) where nb is the number of models that caught bug b. Then DiscF0.5 = 1.25 · P · RareR / (0.25·P + RareR).

Best for teams that: do bug-archaeology on legacy code; want a "second-opinion" model that catches what their primary one misses; build red-team / chaos-engineering tooling; run periodic eval-gap audits to find bugs nobody else has reported yet.

Skip this metric if: you mostly care about catching the obvious bugs reliably (use F_0.5 or Recall instead).

Precision — of every flagged bug, how many are real?

What it measures: The fraction of the model's findings that are real defects. High precision means the engineer reading the report can trust each finding without hours of triage; low precision means most of the output is noise. Counts every false positive equally regardless of severity. FP = total false flags, hallu = the subset the second-pass judge classified as outright fabricated (not real bugs at all).

Math: P = TP / (TP + FP). Range 0–100%. A model that emits one finding and is correct scores 100%; a model that emits 100 findings of which 90 are real scores 90%. Indifferent to recall — a model that catches 1 real bug out of 50 still scores 100% if it had no false alarms.

Best for teams that: feed AI output directly to humans (PR comments, customer-facing reports, compliance audits); run AI in “suggest mode” where each suggestion costs reviewer attention; prioritize trust-in-output over coverage.

Skip this metric if: missing a bug is more expensive than chasing a false one (use Recall), or you have any human review at all (F_0.5 already handles this trade-off well).

Recall — of every real bug, how many were caught?

What it measures: The fraction of seeded ground-truth bugs the model found. High recall means nothing slips through; low recall means real bugs are silently missing from the output. Indifferent to false positives — a model that flags every page element as buggy still gets perfect recall if it caught all the real ones.

Math: R = TP / (TP + FN). Range 0–100%. TP = real bugs caught, FN = real bugs missed. Trivially gameable by emitting every possible finding (so we report it alongside Precision — the two together tell the full story).

Best for teams that: do pre-launch security audits (one missed XSS >> 100 false alarms); accessibility compliance (ADA / WCAG audits); chaos engineering and pre-mortem testing; any “find everything I might have missed” workflow where a human reviewer is going to triage anyway.

Skip this metric if: false positives have any meaningful cost in your workflow (use F_0.5 or Precision).

Groundedness — what fraction of findings is grounded in something real?

What it measures: Of every finding the model emitted, what fraction is grounded — backed by something real on the page — rather than fabricated? Higher is better.

How we compute it: A model’s findings split into TP (matched a seeded ground-truth bug) and FP (didn’t match anything seeded). Each FP is then reviewed by a second-pass judge and classified as either real-FP (a real defect we forgot to seed) or hallucinated (the model invented it). Hallucinations are a strict subset of FP, so they can never exceed the total finding count.

Math: total findings = TP + FP. Of those, the grounded ones are TP + real-FP; the rest are hallucinations. So Groundedness = (TP + real-FP) / (TP + FP) = 1 − hallu / (TP + FP). Range 0–100%. A model that emits 20 findings with 1 hallucination scores 95%; a model that emits 5 findings, all real, scores 100%.

Use this metric when — you want to avoid a false positive at all costs. Best for: feeding AI findings to automated downstream systems where there is no human gate — auto-PRs, auto-fix bots, ticket auto-creation, alerting rules. A single hallucinated finding becomes a fake regression, a fake ticket, or a 3 a.m. page; one fabrication can cascade into production breakage.

The trade-off: optimizing for groundedness comes at the cost of discovering more issues. The strictest mode keeps only findings that pass multiple corroboration filters — fewer hallucinations, but real bugs that only one leaf flagged get dropped too. If you want maximum coverage, use Recall; if you want a balance, use F_0.5.

Skip this metric if: a human reviews every finding before action (then F_0.5 or Precision is the right metric), or you specifically need to find every real issue (use Recall).

Reading this leaderboard fairly: a model can score 100% groundedness simply by emitting very few findings — if you only flag 3 things and all 3 are real, you’re trivially perfect. The bare models tied at 100% below typically catch 2–7 bugs total across the cohort. testers.ai (groundedness) catches ~36 real bugs at 92%+ groundedness — near-perfect, but with about 10× the coverage. When choosing a model for an automated pipeline, consider both columns together: groundedness and the absolute count of findings.