Harnesses find more verified bugs, and open-weight models make them cheap: a systematic evaluation of AI code review - prompts, models, effort levels, and harnesses (September 2026)

Developer edition Executive summary Slides Author disclosure Reproduce it Data and code

In one line: On six severity-selected PRs from two codebases, harnesses increase true-gold recall over the same model’s vanilla prompt in 39/42 comparisons (7 models × 2 harnesses × 3 efforts; Fable excluded because its harness coverage is incomplete). Opus · CE · medium leads the revised advisory-reward F2′ point estimates; GLM harness cells offer lower-cost pilot options. This is evidence for a pilot, not production certification or statistical equivalence.

Abstract

This evaluation compares bugs found, review noise, API cost and latency across single-prompt and agentic code review. It provides executable hidden-defect evidence and paired comparisons to help teams choose a configuration to pilot. Eight models (three Claude-family, three OpenAI-family, two open-weight GLM) ran through three frameworks — one-shot prompting plus two agentic harnesses — at three reasoning-effort levels on the six highest-severity PRs of a public 50-PR benchmark: 2,416 healthy runs, 66 of 72 complete model×framework×effort cells. Quality is scored under two lenses: the benchmark’s 42 human-verified golden comments, and a primary true golden set built by auditing the campaign’s own discoveries into 105 verified hidden defects with archived reproduction/fix-test evidence (147 distinct bugs; a September 2026 audit withdrew three and merged two duplicates). Unmatched findings carry saved bug, important-non-bug and unsupported judgments; headline numbers carry PR-level cluster-bootstrap intervals, and cost, tokens and wall-clock are metered.

Figure 1 — Value for money: what a dollar per review buys interactive

How to read: each point is one complete cell (a model × framework × effort combination); the x-axis is metered dollars per PR review (log scale) and the y-axis is F2′, our advisory-reward composite alongside recall on the 147-bug true golden set. Lines trace each model’s effort ladder within a framework; hover for the configuration and values.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none fable-5.1 astra sol opus-5 glm-vis terra sonnet-5 glm-flash

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

show 95% CIs

Static fallback (used automatically when JavaScript is unavailable)

Harnesses improve true-gold recall over the same model’s vanilla prompt in 39 of 42 matched comparisons (7 models × 2 harnesses × 3 efforts; Fable excluded), with mean Δrecall +0.135 and a peak-recall ratio of 1.63×. The revised F2′ rewards useful accepted advisories alongside verified bugs; §2.6 discloses its weights and classifier limitations, and §3.1 reports current rankings and sensitivity. Low-effort GLM harness cells offer inexpensive pilot options. Medium/high GLM effort can incur substantial latency.

Caveats: one run per cell per PR (intervals cover PR sampling only); a severity-weighted six-PR sample; instrument-sensitive precision. Read the intervals, not the rankings.

1. Introduction

Why this report exists. Teams choosing automated code review need to know whether extra model passes find more defects, how much noise they add, and what they cost per review. We compare the same models under one-shot prompting and two harnesses, with audited defect evidence and metered costs.

What we measured. The campaign covers a public 50-PR benchmark across five codebases, while the primary 147-bug analysis covers six severity-selected PRs from two codebases. Its hidden-defect portion has 105 fail-on-head/pass-on-fix tests. Results describe these workloads and configurations; developer time saved, deployment reliability and performance on other repositories remain unmeasured.

The questions.

1. Q1 — Do harnesses find more real bugs than one-shot prompting? If not, the orchestrator premium buys nothing; if so, how large is the gap, and does it hold across models and efforts?
2. Q2 — Which framework, effort and model cell should a team actually run? The practitioner’s choice is among cells, so the results are organised around cell-level cost/benefit.
3. Q3 — What do those cells cost in money, tokens and wall-clock? Measured at published list prices with a retrieval date, cache pricing included — the folklore ratios above show how easily cost claims drift.
4. Q4 — How much of the apparent signal survives evidence audit and instrument comparability? A cluster is not a bug, an unmatched finding is not necessarily a hallucination, and a metric that cannot be computed for some cells is not a zero.

Scope and history. One selected run per cell per PR — PR-level uncertainty is quantified by cluster bootstrap, but repeat-run variance is not measured. One benchmark, one 50-PR sample, headline numbers from its six highest-severity PRs (§2.4 tests that this is a hard-but-valid slice, not a recall-harder one). This report supersedes the August 2026 experiment (archive/report-2026-08-26-batch083.md), which scored the same frameworks against human labels only; what is new here is executed, audited hidden-gold ground truth and per-review metered cost.

In brief. Under the strict 42-golden lens, the recommended open-weight cell (GLM-5.3, run under the gateway route name glm-5.3-vision-background; metareview, low effort) had the same golden-recall point estimate as opus CE-low (0.81 each; paired ratio CI 0.91–1.09) and a higher F1 point estimate at 7.5% [6.7–8.5%] of the opus compound-engineering cost; the cheaper GLM-5.3-Flash (glm-5.3-flash-background; both names are explained in §2.2) reaches recall 0.83 [0.74, 0.92] at 0.8% of the opus CE-low cost, paying in precision (adjP 0.54 vs 0.67). Under the true-set lens, harnesses improve recall over their own model’s vanilla baseline in 39/42 matched comparisons (mean Δrecall +0.135). Read §3.1 for the advisory-reward F2′ comparison, which is a revised preference rather than a change in bug discovery. The highest-recall cell finds 88 of 147 true bugs and misses 59; other complete configurations recover 52, leaving 7 unfound (the union covers 140/147).

How to read this report. §2 is the method — what we ran, how findings are judged, why six PRs, and how we built ground truth we could execute; read it if you want to check or reproduce a number. §3 is the results, in five claims plus the cost/token/latency economics and a check that the six-PR sample is not misleading. §4 states plainly what we cannot claim, and §5 gives the exact reproduction chain (inputs, commands, checksums). If you only read one thing, read §3.5 (the five claims); if you only look at one figure, look at Figure 1 (value for money). Appendix A keeps the superseded unions for provenance only.

2. Methodology

2.1 The lab and the benchmark

Benchmark-defined (Martian offline code-review benchmark, README third_party/code-review-benchmark/offline/README.md): 50 PRs across 5 codebases; 173 human-verified golden comments with severity (Low/Medium/High/Critical) and category tags. Pipeline: extract → deduplicate → LLM-judge semantic matching → profile-based scoring.

• Scoring profiles: Strict = {bug, security, concurrency, data, api} (139 goldens); Core (default) = +{perf, test_gap, doc_defect} (158); All = +{style, speculative} (173). A candidate matching a golden excluded by the active profile is matched-excluded — neither rewarded nor penalised. Our frozen matcher matches against all 173 goldens, so stored recall is the All profile; Core/Strict splits are derived per run from per_golden_matches. On the severity top-6, the three profiles differ by at most one golden (PR 10967’s single style comment), so top-6 numbers are effectively profile-invariant (§3.2, T1).
• F-beta: F1 and F2 (recall-weighted 4:1 — a declared preference, not measured cost) are reported for every cell. Fβ = (1+β²)·R·P/(β²·P+R).
• Judges: gpt-5.2 for the Anthropic-model rows (fable/opus/sonnet) and the GLM rows; claude-opus-4-5-20251101 for the OpenAI rows (sol/terra/astra). Anti-self-preference by vendor, single judge family per row (so within-row framework comparisons are judge-consistent); cross-judge agreement is only partially measured (§4, analysis/ADJUDICATOR_INTERRATER.md).

Our extensions beyond the benchmark (each with its tool, each disclosed per-metric):

• Honest re-adjudication (precision side). The benchmark’s raw precision counts every unmatched candidate as a false positive; Martian’s own methodology concedes the golden set is incomplete. We adjudicate every unmatched finding three-way: bug / important_non_bug / hallucination (v2 in-run, harnesseval/adjudicate.py; hardened v3.1 = readjudicate3.py with cross-run clustering, grounded hallucination, provenance stripping, k-vote). adjP = TP/(TP+hallucinations); beyond-gold = bug+important_non_bug outside the golden set. Original campaign judgments remain frozen, including rj3 at k=1 with v3.1 clustering. This revision adds a separately archived common advisory re-adjudication (§2.6); it preserves original bug credit and legacy metrics. Section 2.3.3 describes the original instrument effects.
• Cluster (PR-level) bootstrap CIs on every headline number (§2.3.2).
• Metered cost accounting at published list prices (§3.4) with a retrieval date, pricing fresh input / cached reads / cache writes / output separately.
• Analysis docs we build on and do not rewrite: analysis/ADJUDICATOR_INTERRATER.md (adjudicator agreement 0.43–0.83 across judge models; the v1→v3 instability motivation), analysis/ADVISORY_RECALIBRATION.md (advisory-channel accounting fix; mrv 0.11.1–0.11.2), analysis/CE_VS_MRV_GAPS.md (15% CE-only hidden-gold; ~9% verdict flips across phrasings), analysis/EXTERNAL_REVIEWER_GAPS.md (92% of CodeRabbit/BugBot findings covered by our union), analysis/ACCEPTANCE_0111_FINAL.md (0.11.1 acceptance with k=3 adjudication), analysis/CE_RESIDUE_AFTER_UPGRADE.md (hidden-gold gap decomposition), FURTHER-RESEARCH.md.

2.2 What we ran: frameworks, models, efforts, judges

The matrix. 8 models × 3 frameworks × 3 reasoning-effort levels = 72 cells, a cell being one run of one combination on one PR. On the six-PR primary sample 66 cells are complete (396 runs, n = 1 run/PR); six fable compound/metareview cells have 1–2 PRs — coverage gaps, not results (analysis/COVERAGE.md). The campaign holds 2,416 healthy runs, 2,234 of them selected (one per cell×PR); run health and the era rule deciding which batches a framework may draw from are in §2.3.2.

Models. Three vendor families plus open weights:

A note on the GLM names, and on what was actually served. glm-5.3-vision-background and glm-5.3-flash-background are the route names on the gateway the GLM runs went through (LunaRoute; see Special thanks). They are kept verbatim so that every row can be matched to the run registry. The -vision route serves lunaroute/GLM-5.3-Vision-NVFP4. Its model card describes it as the GLM-5.3 text backbone taken unmodified from RadixArk/GLM-5.3-NVFP4, an NVFP4 (4-bit floating point) quantization of Z.ai’s open-weight GLM-5.3, with a vision tower and projector added. Every request in this study was text only (diffs and code; no adapter sends image input), so the vision components were never exercised. The GLM-5.3 results in this report are therefore results for a 4-bit quantized build, not for the full-precision weights; another build, or another host’s deployment, may score differently. As we understand it from the provider, -background selects a routing option on the gateway and not a different model. We did not independently verify the gateway’s serving configuration, and the build behind the -flash route is not documented here. We call the two models GLM-5.3 and GLM-5.3-Flash. Where a table or sentence below says “vision” or “GLM vision”, read GLM-5.3; where it says “flash” or “GLM flash”, read GLM-5.3-Flash.

Frameworks. vanilla-engineered is an engineered single prompt (8-category rubric plus severity guidance) executed as one model call with no subagents — the baseline of pointing a coding agent at a diff. compound-realistic (CE) drives the Compound Engineering ce-code-review skill: a risk-driven persona roster dispatched as parallel subagents, then a separate synthesis pass. metareview-realistic (MRV) drives the metareview binary: deterministic Go gates (free) plus eight adversarial LLM lenses as parallel subagents, single-pass synthesis. Both harness adapters run the real plugin/CLI inside a host agent loop, so subagent dispatch is real — with cost-attribution consequences disclosed in §2.3.

The one-shot baseline prompt (verbatim)

The vanilla-engineered baseline is not a bare “find the bugs” request. Every one-shot cell sends the prompt below as a single model call, with the pull request’s title and diff substituted for {pr_title} and {diff}. It fixes a reviewer role, the eight issue categories of the benchmark’s Core scoring profile (§2.1), a per-finding severity, an instruction to report only confident, real issues, and an output format. It is defined as ENGINEERED_PROMPT in harnesseval/adapters/vanilla.py; the deliberately minimal NAIVE_PROMPT in the same file is not part of this matrix.

You are an expert code reviewer. Review the following code diff for real, actionable issues.

PR: {pr_title}

```diff
{diff}
```

Find issues in these categories: bug, security, concurrency, data, api, performance, test_gap, doc_defect.
For each issue:
- State the specific problem concisely (one issue per item — do not bundle).
- Note the file and line if identifiable from the diff.
- Classify severity as Low, Medium, High, or Critical.
- Only report real issues you are confident about; do not pad with style nits or speculation.

Respond with a numbered list, one issue per line, e.g.:
1. [High/bug] path/to/file.py:71 — description of the specific problem
2. [Medium/performance] ...

Effort. low / medium / high map to the provider’s reasoning-effort setting (Anthropic: thinking disabled at low/medium; OpenAI/GLM: reasoning_effort); xhigh is outside this matrix (§3.5.4).

Original campaign judges and adjudication instruments. Golden matching and three-way adjudication (bug / important_non_bug / hallucination) used cross-family judges to reduce self-preference risk: gpt-5.2 for Claude- and GLM-family rows, claude-opus-4-5-20251101 for OpenAI-family rows, k = 1 (campaign lock). Three precision instruments appear across cells — v1 (legacy binary, unable to classify nitpick-class findings), v2 (in-run three-way), rj3 (v3.1 clustered re-adjudication); their comparability limits are disclosed in §2.3.3 and continue to qualify the legacy precision/burden scores. The new F2′ uses a separate common GLM-base re-adjudication instrument (§2.6), whose same-family judge risk is explicitly retained.

Samples. Primary: the six PRs with the highest summed original-golden severity weight (§2.4). Secondary: the full 50, used only for the selection-effect check on the 33 cells with ≥40/50 healthy scored PRs (§3.6). All headline numbers are primary-sample numbers unless labelled otherwise.

Caveats: what we did not run, and why

One gap is a decision of ours, and it is worth stating plainly rather than leaving it implicit: we did not run Claude Fable 5.1 across all PRs and all harnesses. Fable 5.1 was run one-shot across all six primary PRs, but its two harness cells (compound-realistic and metareview-realistic) were run on only one to two PRs each. Every other model has its full harness grid on the primary sample.

The reason is budget. A harness review is not one model call: the orchestrator dispatches several subagent passes and then synthesises, so a harness run costs many times a one-shot run — and Fable 5.1 is a frontier-priced model. A full Fable 5.1 grid across the PRs and all three frameworks was more than this campaign could fund, and we judged the same money better spent on depth everywhere else: executed ground truth, honest adjudication, and the full-50 comparison runs. The partial Fable harness numbers are therefore gaps, not findings, and no claim in this report rests on them.

We will amend this report with updated numbers in either of two cases:

• A sponsor funds the runs. The missing Fable 5.1 harness cells can be run on the same apparatus and merged into the same tables; reach out via the repository (github.com/dsifry/harnesseval).
• Someone runs them and sends us the data. The methodology is laid out to be reproduced: §2 specifies the frameworks, models, effort levels, judges, instruments, run-health and era rules, and §5 publishes the reproduction chain (runs/ inputs, the exact commands and the checksums). Data produced that way drops into the same scoring code and the same tables, keeping the comparison like-for-like. Send it to us and we will amend the report with it.

Nothing in the conclusions depends on guessing what those cells would have shown: every headline result is computed from the cells that are complete, and analysis/COVERAGE.md records exactly what is missing.

2.3 How findings are judged: instruments, comparability, and disclosure

2.3.1 The judging instruments, frozen across the campaign

harnesseval/judge.py (golden matcher), readjudicate3.py prompt/version semantics, and the golden dataset were not modified for this report. n = 1 selected run per cell×PR (selection rule in §2.3.2); duplicate healthy scored runs exist for some vanilla cells (listed in analysis/COVERAGE.md) and were not pooled. Adjudication k = 1 (campaign lock); if anyone re-adjudicates at k≥3, both numbers must be reported (analysis/ACCEPTANCE_0111_FINAL.md shows what k=3 looks like on the 0.11.1 acceptance cells).

2.3.2 Run selection and the era rule

One healthy, scored run per (model × framework × effort × PR): health rule = no error, tokens>0, findings-or-≤20k-tokens; scored = integer tp/fn present; era rule (banked 2026-09-13): vanilla-engineered pairs accept healthy runs from any batch/era; compound/metareview pairs must come from batch 20260910-mrv0120-manifold (pre-0.12 metareview binaries are instrument-confounded). Within the rule we prefer the current campaign batch, then the fable vanilla batches, then newer legacy batches (tools/final_report_extract.py).

2.3.3 Precision-instrument disclosure (per cell, in T1)

rj3 files exist for ~half the vanilla cells (opus/sonnet vanilla, fable vanilla low, parts of astra/sol/terra/glm-flash) but almost no compound/metareview cells of the current campaign; the harness cells use the in-run v2 three-way adjudicator (same judge family, same taxonomy, k=1, per-finding instead of clustered). Measured on the same runs, the v1 binary instrument (pre-2026-09-07 vanilla reuse) over-counts hallucinations massively — e.g. opus-5 vanilla low adjP v1 0.50 → rj3 0.90 (28→3 hallucinations), fable vanilla low 0.46→0.82 (37→7) — which is why v1 cells (fable vanilla medium/high, opus vanilla medium) are flagged in T1. These examples show that the older instrument can depress scores; they do not establish a quantitative bound or the bias direction for unmeasured cells. The v2→rj3 gap is the residual instrument risk for harness cells; analysis/CE_VS_MRV_GAPS.md measured ~9% verdict flips across phrasings under v2 that v3.1 clustering removes by construction — direction of bias on aggregate adjP is not established.

2.3.4 Provider incident disclosure (GLM rows)

The 2026-09-14/15 Lunaroute gateway incident (aggregate concurrent output > ~400k tokens wedging the batch; OpenAI SDK max_retries=2 silently tripling timeouts) was fixed by ~2026-09-15 16:00 (max_retries=0, first-attempt retry + streaming, lens semaphore 4, FD headroom). Per-cell exposure, using a conservative cutoff of 2026-09-15 23:00 UTC (16:00 PDT): every GLM harness cell on the top-6 contains pre-fix runs (glm-vis mrv medium 5/6, high 2/6 are the least exposed); the glm-flash vanilla cells selected the post-fix 2026-09-16 runs. The incident was operational (reliability), not a prompt or dataset change, and every selected run passed the health gate; but the headline GLM cells (incl. the §3.6 recommendation cell glm-vis MRV low, run 2026-09-10) cannot claim post-fix infrastructure. Treat GLM quality numbers as carrying this caveat; the gateway-capacity caveat (throughput, not quality) is in §3.5.5.

Also disclosed: the six glm--compound cells and glm-vis MRV low lack per-model token splits (per_model_usage); their metered $ prices all* input at the fresh rate (no cache discount), so those cells’ $ are upper bounds on the input side. Anthropic rows carry real provider billing (cost_usd), which our metered table reproduces to within 4% for opus/sonnet (median metered/billed 0.96/0.96) and underestimates fable-5.1 billing by ~15% (median 0.85 — cache-write pricing), so fable $ are conservative.

2.3.5 Author interest disclosure (metareview)

One of the harnesses evaluated in this report, metareview (MRV), is authored by the author of this report. metareview is open source, MIT licensed, and available free of charge at https://github.com/dsifry/metareview. The interest is disclosed because MRV appears throughout the headline comparisons (§3.1, §3.3–§3.5). The safeguards that limit its effect on the reported numbers: every harness is judged by the same frozen instruments (§2.3.1); judging is vendor-anti-self-preference, and within-row framework comparisons are judge-consistent (§2.1); the true-gold set is built from executed failing tests and fixes, not from harness output (§2.5); and the matched framework comparisons use the same model/effort mix (§3.3). Readers should weigh §3.5.5’s recommendation with this in mind, and can re-run the entire offline chain themselves (§5).

2.4 The top-6 design choice, argued and tested

Why six PRs, and why these six. The campaign’s grid is 8 models × 3 frameworks × 3 effort levels = 72 cells, and every cell × PR combination is a complete review — a harness cell being many model calls, not one. Run over all 50 benchmark PRs that is 3,600 reviews. The budget for this study allowed the full grid on a six-PR primary sample (432 cell×PR reviews, 66 complete cells), with the remaining budget spent on depth instead of breadth: executed ground truth for every claimed defect, honest adjudication of every finding, and full-50 comparison runs used to check the sample (§2.4.2, §3.6). In short, the trade was PR breadth against depth and redundancy of evidence, and we chose depth on a hard slice.

The six were selected by summed original-golden severity weight — an objective, pre-registered criterion, not a hand-picked set — and it turns out to be the six PRs with the highest summed original-golden severity in the evaluation set (mean summed severity 18.2 versus 6.1 for the other 44, about 3×; §2.4.2). The six span two codebases. The benchmark-golden checks in the rest of this section show the selection effects and their limits: it is hard-weighted, but recall measured on it is not systematically biased against the tools (§2.4.2), and the earlier hope that it would be a conservative lower bound on full-set results does not survive testing — which is exactly why we report intervals and a selection check instead of treating six PRs as the whole story (§2.4.3, §3.6).

2.4.1 What “top-6” means, and why it is a defensible slice

top-6 = the six PRs with the highest summed golden-comment severity weight (Critical=4, High=3, Medium=2, Low=1; ties by comment count) — i.e. the six PRs with the highest original-golden severity totals (CAMPAIGN_GLOSSARY.md; verifier tools/verify_hitlist.py; artifact manifold_top6_hitlist.csv): cal.com/11059 (sev 26, 9 comments), discourse-graphite/4 (21, 8), discourse-graphite/10 (21, 7), cal.com/14740 (14, 6), discourse-graphite/8 (14, 6), cal.com/10967 (13, 6). This selection is by ground-truth severity density, not by any measured detection difficulty for these tools — §2.4.3 below tests directly that the top-6 is not measurably recall-harder, so read “hard” below as severity-weighted, not as an estimate of what the tools find hard. This is a deliberate weighting toward higher-severity workloads: a code-review tool that only catches easy issues is not worth buying, and cost/benefit is decided at the hard end. (A lexicographic misreading of “top-6” already corrupted one subset earlier in this campaign — the severity ordering is binding.)

2.4.2 Is the sample representative of the full benchmark? (tested)

How hard is the sample? Measured directly (panel 5 of the dashboard): summing each PR’s golden severity weights (Critical 4 … Low 1), the six headlined PRs average 18.2 against 6.1 for the other 44 — about 3× — and the mildest of the six is as severe as the most severe PR we did not pick (both 13). The sample is deliberately severity-weighted; it is neither a random nor a representative sample of production review.

Scope of this check: benchmark-golden metrics only. The 147-bug true-gold set exists only for the six selected PRs; these checks do not establish its generalization. Selection effects were examined two ways on the 33 cells with ≥40/50 healthy scored PRs (T3):

1. Direct gap (selection effect). Mean recall gap (top-6 − full-50) = +0.006 (median +0.014, range −0.14…+0.14; vanilla +0.005, harness +0.007): recall measured on the top-6 is close to full-set recall on average across these observed cells — the severity-weighting does not make the top-6 recall-harder for these tools. The F1 gap is larger and systematic (+0.065 mean, harness +0.084): harness adjP is higher on the top-6 (the non-top-6 PRs elicit more hallucinated noise from harnesses), so top-6 F1s overstate full-set F1 for harness cells and should be corrected by roughly the T3 per-cell gaps when extrapolating.
2. Ranking agreement. Spearman ρ of model F1 ranks (top-6 vs full-50) within framework×effort: 0.80 for vanilla-low, CE-medium, CE-high and MRV-low, and 0.37 for CE-low (T3): the CE-low ranking is the one place the top-6 reorders models (sonnet-5 collapses from 3rd on full-50 to last on top-6); everywhere else ranks are preserved or near-preserved. This is direct, if not uniform, evidence for the claim.

2.4.3 Is the top-6 result a conservative “lower bound”? (No — tested, and why that matters)

What the sub-claim was. If the six chosen PRs really are the hardest in the set, it is natural to hope that scores measured on them are a floor for the rest of the benchmark: a cell that scores X on the top-6 would then score at least X on the full 50, and our headline numbers would be conservative by construction. This is the “lower bound” sub-claim. It matters because it decides how to read every number in §3: as a floor to be improved on, or as a result conditional on the selected severity-weighted slice.

Why it is false. A floor would require recall to fall as PR severity rises. It does not: Spearman ρ between each PR’s severity weight and its recall, over the 33 cells with full-50 runs, averages +0.10, with 6 of 33 negative — recall is not monotonically lower on harder PRs. So the honest framing is that the top-6 is a hard-weighted sample whose recall matches the full set on average (§2.4.2), not a lower bound. Where an individual cell does diverge sharply (e.g. sol vanilla high: top-6 0.62 vs full 0.76), the T3 table is the correction — and that divergence is also why a selection check is reported at all (§3.6).

2.5 Building ground truth we could execute

Why the human-labelled gold was not enough. The benchmark’s 42 golden comments are expert-labelled and well defined, but they are only what those reviewers happened to write — they cannot tell us whether a finding outside the list is real, and a buyer comparing harnesses needs exactly that. The campaign’s own discoveries are the only larger source of candidate truth, and a discovery is only trustworthy if it can be checked mechanically. So the ground truth used in §3.1 is not a label set: it is a set of defects each of which owns an executed test.

From clusters to executed defects. Appendix A’s “211 additional bugs” were still clusters — produced by an LLM merge over candidate keys, so they carried both over-counts (one defect restated several ways) and under-counts (real defects the audit split away and never restored). §3.1 replaces them with verified defects with archived reproduction/fix-test evidence, and the count chain in full runs: 359 raw pre-merge clusters → 253 after the strict re-merge → 152 candidates → 147 distinct after the 2026-09-18 audit (42 goldens + 105 verified defects).

Construction (all artifacts in analysis/verified_gold/). For each candidate the pipeline authored a test, ran it on the PR head (must FAIL), authored a minimal fix, ran it again (must PASS), and — where a sibling defect existed at the same site — re-ran under the bundle’s fix, which must leave it RED (orthogonality: a defect is only counted separately if its own fix is required). The merge audit (MERGE_AUDIT.json) then split multi-concern bundles; an independent six-reviewer duplicate pass plus fix-location adjudication removed 23 defect-level restatements; container-level folding removed 17 more. Finally the audits’ own label lists were walked for under-counts, restoring 10 defects that had been dropped (including an unset-secret auth bypass and a missing scheme check before open(url)). Every defect now owns defects/<id>/{test.diff,fix.patch,logs/,meta.json}; the canonical list is analysis/verified_gold/GOLD_DEFECT_CATALOG.md (110 entries, one per defect — 105 verified distinct, plus 3 marked WITHDRAWN and 2 marked DUPLICATE by the 2026-09-18 audit — each with its test, its fix and its provenance). Provenance is recorded per defect rather than inferred from a count of successful fixes. Among the 110 retained catalog entries, 30 have explicit recorded sibling-fix-red evidence; that cross-fix check is unrecorded for the other 80. These counts include withdrawn and duplicate entries and must not be read as 110 independently verified orthogonality checks. The comparison numbers quoted below (paired deltas, ensemble coverage, the best cell’s misses) are persisted in final_report_metrics.json under true_gold_defects.<variant>.derived.

Verified universe: 42 goldens + 105 verified distinct defects = 147 true bugs. All 105 are D-verified (fail-on-head → own-fix-pass). A 2026-09-18 post-publication audit — triggered by external review — withdrew 3 defects whose tests did not demonstrate the claimed behavior (11059-D28, 4-D04, 4-D29) and merged 2 as duplicates (11059-D32 → 11059-D19; 11059-D14, already described by the original golden); see analysis/verified_gold/WITHDRAWALS_AND_DEDUP_2026-09-18.md. What the audit taught us. Three pipeline failures were found by adversarial re-checking, and each one changed the numbers:

• A test can manufacture its own failure. One candidate’s test mocked the very dependency whose behaviour the claim was about (a mocked zod with no default export, while the installed package has one); another’s recorded “head failure” was a harness stub error (blank? on a String) that fired before the claimed behaviour was ever reached; a third failed with an unrelated NameError while the claimed exception never fired. Re-reading the failure reason in every one of the 110 head logs — not just the red/green exit status — is what surfaced them, and is why the three defects named above were withdrawn rather than counted.
• Duplicates can hide across containers. Two defects described the same missing-secret guard at the same call site, differing only in the HTTP status the fix returns; a third restated a mechanism the original golden already names, fix site included. Candidate pairs were adjudicated with the repo’s cross-fix standard — evidence that neither defect’s own minimal fix cures the other’s test supports keeping the pair distinct; shared mechanisms and fixes support merging — which merged 2 and kept 4 borderline pairs (4-D14, 4-D45, 11059-D33-vs-D22, 4-D56-vs-4-D36) as genuinely distinct.
• The scoring map itself had a quiet hole. The finding→defect assignment considered only the first 60 findings of each multi-defect bundle: 276 findings were silently dropped (241 findings across 51 credited cells), and at least one finding was mis-credited (the embed_by_username.downcase-on-nil finding went to the queue-flooding defect 4-D04 instead of 4-D56). The repaired pipeline considers all 2,923 findings, permits explicit no-match nulls (71 of them), and re-verifies low-signal assignments (DEFECT_ASSIGN_AUDIT.json). Every §3.1 number was recomputed from the repaired inputs — which changed the absolute bug-recall and then-current composite levels.

The audit’s full record, with the evidence behind every decision, is analysis/verified_gold/WITHDRAWALS_AND_DEDUP_2026-09-18.md.

One honest boundary: the 105 are the defects the pipeline could execute. The earlier audits’ label lists suggest a small number of further claims whose tests did not converge; we report those as in-doubt plumbing rather than as verified bugs (UNDERCOUNT_2026-09-18.md).

2.6 Metrics and the evaluator choice

Three choices shape every number in §3: how defects are counted (the metrics tool below), which composite is read (F2′), and which cells are allowed to be compared at all (the instrument caveat). Each is a stated choice with a cost, and each is stated here rather than buried in the results.

What a 95% confidence interval (CI) means here. Quality metrics are pooled ratios of summed PR-level counts, not averages of the six PR scores; cost and latency are per-review means. We resample the six PR clusters with replacement 10,000 times, recompute each statistic, and report its 2.5th and 97.5th percentiles. These intervals describe sensitivity to resampling the observed PRs. They are not prediction intervals for a new six-PR dataset, and severity selection limits population inference. Overlapping marginal CIs do not establish a tie, equivalence or absence of a paired difference: paired comparisons use the CI of the difference (or ratio). A difference is called resolved when its paired CI excludes zero. Intervals omit run-to-run model variation (one selected run per cell per PR), uncertainty from choosing the best cells after seeing the results, and unmeasured judge error.

Metrics: tools/verified_gold_defect_metrics.py (cluster bootstrap, B=10,000, seed 20260916); finding-to- defect assignment by tools/verified_gold_defect_assign.py (repaired 2026-09-18: every one of the 2,923 findings is now considered — the pre-repair version silently dropped 276 past a 60-finding cap; audit trail in DEFECT_ASSIGN_AUDIT.json); the section is also exposed as true_gold_defects in analysis/final_report_metrics.json.

Which composite to read. F2′ now rewards useful staff-level review advisories as well as verified bugs. This revision changes the evaluator preference and adds a common adjudication pass; it generates no new review runs and does not measure developer time. The bug-only F2 and recall remain unchanged. For each cell we pool:

• T: distinct credited bugs from the 147-bug verified universe (D = 147).
• A: accepted useful important_non_bug advisories from the original unmatched finding texts, classified through the common re-adjudication pass and deduplicated per run using validated identities for the selected scoring policy. Findings already assigned to a verified bug are excluded from all advisory, penalty and below-threshold counts: their frozen verified-bug assignment is authoritative. The original classifier decision and verified bug IDs are retained for audit.
• H: effective unsupported findings. False claims, style-only comments and vague speculation carry the same unit penalty when the current classifier assigns hallucination with confidence ≥0.80. Findings classified as useful advisories do not enter H, even when below the advisory-credit cutoff.

The primary score is F2′ = (5T + αA) / (4D + T + αA + H), with α = 1. An additional accepted advisory increases this score; an unsupported finding decreases it. A missed verified bug retains four units of denominator weight. This is an advisory-augmented composite, not an ordinary F-beta score computed from advisory recall. We report α = 0.5 and 2 as sensitivity checks. The choice of α expresses our preference for useful review breadth and has not been calibrated against human triage time, downstream fixes or economic value. Unbounded advisory volume can drive the score toward one, so the acceptance bar, deduplication and separate T/A/H counts are essential to interpreting it.

What qualifies as an advisory. The common-pass prompt includes the four explicit criteria from tools/enum_advisory_ceiling.py: true, decision-relevant to a staff maintainer, not a correctness bug and not style, with a concrete benefit to the maintainer. It retains the existing adjudication category system while preserving historical rj3 defaults separately. False claims, pure style/format nits and concerns too vague to act on share the hallucination penalty category. An unverifiable but plausible concern retains its classification and receives neither credit nor a penalty when below its scoring threshold.

Advisory credit and unsupported-finding penalties are separate decisions. Revised F2′ has no penalty merely for reporting a non-bug. A finding classified as a useful advisory can earn positive credit or remain unscored; falling below the advisory cutoff never turns it into a penalty.

The classification comes from the current common pass, not necessarily the finding’s original review label. The hallucination category includes false factual claims, style-only nitpicks and vague, non-actionable speculation; it does not mean every penalized finding contains a fabricated fact. Frozen verified-bug assignments take precedence over model labels, preventing double credit or a contradictory penalty. These confidence values are model-reported judgments, not calibrated probabilities or measured advisory value. All below-threshold findings remain classified, with their category and confidence retained for audit.

Common pass methodology (all six PRs validated). The selected judge is glm-5.3-background at low effort, k=1, initially with four concurrent calls, processing the top six PRs strictly one PR at a time. The selected report policy uses confidence ≥0.70 for advisory credit and ≥0.80 for penalties, both inclusive. Thus an important_non_bug judgment at 0.75 earns credit, while a hallucination judgment at 0.75 receives no penalty. Below-threshold findings remain classified but unscored; their original category, response, reasoning and confidence are preserved. The raw pass’s original 0.80 export remains frozen; analysis/verified_gold/advisory_readjudication/active_scoring_policy.json identifies the separately versioned report policy. Newly eligible advisories require policy-specific duplicate checks, reusing existing pair judgments where valid. Penalty identities and the 0.80 equivalence threshold are unchanged. Classifier bug labels never create verified bug credit. Already admitted verified-defect identities reuse audited truth per member without another classifier call or an invented confidence. The choice followed matched low/high pilots, documented in analysis/verified_gold/advisory_readjudication/MODEL_PILOT_COMPARISON.md; those pilots establish neither calibrated confidence nor equivalent intelligence or production accuracy.

A separately recorded throughput trial increased concurrency from four to six during final duplicate validation on PR #14740. In-flight calls were drained before restarting; completed judgments and frozen request/proposal identities were reused. This changes execution scheduling, not the judge, prompts, thinking level, scoring thresholds or verified bug truth. The trial and four-call baseline are archived under passes/glm_base_claim_v3/execution_extensions/concurrency6_v1/ in the advisory readjudication directory. In equal five-minute final-pair windows, four calls completed 840 comparisons (168/minute), versus 500 (100/minute) at six after a 60-second warmup. Completed-request median latency rose from 21.45 to 45.11 seconds; schema retries were two versus zero. Six recovered to 200 comparisons/minute in its last minute, so this short, different-batch comparison does not establish a causal capacity limit. Four remains the selected default; eight was not trialed. See result6_5min.json in that archive for the measurement windows, request hashes and caveats.

The subsequent policy-specific advisory deduplication uses a separately archived execution_extensions/request_timeout_v1 wrapper: each router call has a 120-second wall-clock deadline, including internal retries, with at most three policy attempts. Timed-out child processes are terminated and reaped before retry; completed checkpoints are reused. Queue time is excluded from this deadline. Model, prompts, thinking and scoring rules remain unchanged. Request events record elapsed time and failures; a timeout is a processing failure, never a finding classification. Provider-side cancellation and token usage for terminated requests cannot be guaranteed.

The inventory contains original unmatched finding texts regardless of earlier labels. Each distinct exact cleaned claim, preserving case, receives its own judgment unless audited defect truth is reused. Semantic duplicate counting is a separate step: a shared proposed group does not let one member’s judgment establish another member’s truth. The base pass has validated 403 selected saved runs across all six PRs and 72 cells (66 complete cells contribute 396 runs; seven runs belong to partial cells), including 10,057 independent claim classifications and 101 reused audited bug identities. The ≥0.70 advisory duplicate pass has also completed across all six PRs: 1,501 eligible advisory claims form 888 validated semantic identities. All 403 reviews have validated policy outputs, and all 66 complete cells are eligible for F2′ comparison. Earlier Vision-classifier passes and pilots are not the current instrument.

Reuse and grouping. Fine-grained audited DEFECT_ASSIGN identities override 17 broad older multi-defect groups. Saved semantic coverage spans 7,531 of 12,555 emissions (60%), but old groups and paid votes are candidate evidence only: known overmerges and unsupported consequence variants prevent treating this as fixed identity coverage or 60% classifier-call savings. Grouping uses full cleaned finding text rather than shortened excerpts. Validated file paths and intervals prioritize candidate pairs; overlap does not automatically merge them, different known files remain separate by default, and missing paths remain unknown. Proposed groups undergo a second-stage pair check for agreement in the full material mechanism, trigger and consequence. Uncertain pairs stay separate. Counting groups are split by effective classification so a semantic grouping cannot share false credit across conflicting judgments. Per-member audited truth and model/grouping provenance are retained. Independent final_calculation_validation.json confirms all 403 measured reviews, exact six-PR coverage of the 1,501 claims and 888 identities, every classification source hash, and unchanged values for all 216 cell/universe variants of the frozen bug and legacy metrics. No failed or missing model request is silently treated as an empty advisory result.

Classified findings below the applicable scoring threshold receive neither advisory credit nor a penalty and are disclosed separately; an absent completed classification is not evidence of no advisories. High-confidence classifier bug groups without a frozen assignment to the verified universe are recorded as advisory_uncredited_bugs: they receive no new T or A credit and no H penalty. A mistaken bug label can therefore under-penalize a false claim without increasing recall. Legacy JSON field names advisory_unresolved and F2p_unresolved_bounds remain compatibility aliases for below-threshold counts and hypothetical sensitivity bounds, not processing status. The bounds hold T fixed and compare penalizing all below-threshold groups with crediting them all as advisories. These are deliberately extreme sensitivity scenarios, not confidence intervals, calibrated probabilities, or assertions that all those findings are actually advisories.

The threshold choice is supported by an exploratory 36-advisory, framework-balanced diff audit, with nine claims in each confidence band and 11 additional penalty spot-checks. The classifier metadata was hidden during review, but original claim text sometimes contained the reviewer’s own confidence. The sample informed the selected 0.70 advisory policy; it does not establish a calibrated optimum; A ≥0.80 sensitivity now uses the same ≥0.70 validated identities, with H ≥0.80 and T/D held fixed (§3.1). This isolates the credit cutoff; it is not the earlier grouping or classifier score. An A ≥0.60 comparison was not completed and is not claimed. See analysis/verified_gold/advisory_readjudication/threshold_evaluation_20260919/EVALUATION.md.

This is new model adjudication, not staff-engineer validation. Diff-only evidence limits what the judge can verify. Its archived reasoning and confidence are audit evidence, not calibrated probabilities of correctness. Single-vote judging and a judge from the same model family as two evaluated models leave judge error and self-preference unmeasured. An earlier Vision-judge purposive seven-probe pilot, documented in analysis/verified_gold/advisory_readjudication/CLASSIFIER_PILOT_AUDIT.md, exposed a false active-save claim receiving a raw bug verdict at 0.55 despite the reasoning acknowledging no UI caller. The report policy gives that 0.55 bug judgment no credit or penalty; the probes are not an accuracy estimate or evidence that all false claims are caught. New outputs are separate from original campaign judgments. The reference advisory set covers only two of the six current PRs, so we make no advisory-recall claim and do not add advisories to the 147-bug denominator. Semantic clustering reduces repetition but does not establish that every remaining advisory is independent or that every merge is correct.

Legacy metrics are retained for provenance. adjP′ = T/(T+H_legacy+I_legacy) and the original F1′ charge every important non-bug (I_legacy) as a burden, using their original H_legacy accounting. They are labelled legacy non-bug-burden metrics, not alternative versions of the new advisory-reward evaluator. The former F2′ is retained as F2p_legacy in machine outputs. adjP and bug-only F2 retain their existing meanings; neither rewards advisory breadth. Changes in the new F2′ must not be interpreted as changes in verified bug discovery. Relative to F2p_legacy, both the formula and the judgment/deduplication instrument have changed; the whole score shift cannot be attributed to advisory credit. F2p_without_advisory_bonus sets α = 0 using the same new T/D/H, and F2p_advisory_bonus_lift isolates the bonus relative to that baseline. This baseline is not the frozen legacy bug-only F2.

Instrument caveat. The older v1 binary adjudicator cannot measure important_non_bug; its legacy zero is unmeasured. The common classifier supplies a separate advisory instrument, allowing a current F2′ comparison when classification coverage is complete even if the legacy run used v1. Missing new classification coverage must still be flagged and excluded from headline F2′ rankings. Original adjudication instruments continue to qualify legacy precision and burden scores. Neither new advisory labels nor adjudicator opinions change the test-verified bug universe or bug credit. Bootstrap intervals reflect observed PR sampling, not classifier uncertainty or uncertainty in the choice of α.

3. Results

3.1 The primary matrix: the true golden set

The true-golden ground truth built in §2.5 gives the results below on the six-PR sample; F2′ is defined and justified in §2.6, including the missing-advisory instrument caveat.

Headline (true set, advisory weight α = 1). The common re-adjudication provides measured advisory coverage for 66 complete cells. Rankings exclude any cell lacking completed coverage.

The leading F2′ point estimate is Opus · CE · medium at 0.641. Bug recall retains its own leader, Opus CE-medium, at 88/147 (0.599). Accepted advisories contribute separately to the composite; an improved F2′ does not establish that another bug was found. New classifications do not change bug credit.

The best eligible vanilla cell is Fable · vanilla · medium (0.427). The best-harness-minus-best-vanilla paired difference is 0.214 [0.116, 0.275], with ratio 1.502 [1.215, 1.819]×. Both cells were selected after observing the results, so this comparison is optimistic and does not establish performance on new repositories. PR-bootstrap uncertainty omits classifier and run-to-run error.

Framework comparison. MRV exceeds CE in 18/21 matched point estimates; 95% paired intervals favor MRV in 8/21 cases and CE in 3/21; 10/21 include zero. Mean ΔF2′ is +0.045. The three CE-favoring cases are Opus medium, Sol high and Terra low. On the same 21 model×effort combinations in each framework (excluding Fable from these means), mean F2′ is 0.443648 for MRV, 0.399140 for CE and 0.265356 for vanilla. These are descriptive means of pooled cell scores, not a comparison of unmatched 24-cell versus 21-cell cohorts. Fable remains eligible for configuration rankings. The score within each cell pools PR-level counts. Framework tendencies and the highest single cell answer different questions.

Advisory-weight sensitivity. The leading point estimate at each weight is: α = 0.5: Opus · CE · medium; α = 1.0: Opus · CE · medium; α = 2.0: Opus · CE · medium.

This tests a declared preference, not label uncertainty. The common classifier’s below-cutoff findings receive no bonus or penalty in the primary score; its F2p_unresolved_bounds quantify the separate all-penalty versus all-advisory sensitivity with verified bug credit held fixed. They are not CIs. There is no advisory-recall or developer-time-saved claim.

Confidence-cutoff sensitivity on identical identities. Using the same validated ≥0.70 advisory identities for both columns, fixed H ≥0.80 and fixed T/D, the 21 matched cohorts per framework give:

These are descriptive policy sensitivities, not significance tests or measured developer utility. The full accounting is in analysis/verified_gold/advisory_readjudication/scoring_policies/advisory070_penalty080_v1/FRAMEWORK_COMPARISON.md.

Separate the advisory bonus from instrument changes. For Opus CE-medium, the new common H/T/D with α = 0 gives 0.614525; the 53 credited advisories lift it by 0.026567 to 0.641092. That is the isolated bonus effect. The historical F2′ of 0.460251 used a different judgment and counting instrument, so the entire old-to-new change is not attributable to this bonus.

Unscored findings materially limit interpretation. Opus CE-medium has 298 below-cutoff groups and 60 uncredited classifier-bug groups; vision MRV-low has 139 and 82, and flash MRV-low 148 and 81, respectively. None earns A/T credit or an H penalty. The all-penalty/all-advisory below-cutoff bounds are [0.462, 0.741] for Opus CE-medium, [0.474, 0.639] for vision MRV-low, and [0.462, 0.631] for flash MRV-low. These deliberately extreme scenarios are not confidence intervals and exclude uncredited bug groups. High coverage of processed reviews does not mean that every finding received a scored or correct judgment.

The frozen bug-accounting input also contains three finding-to-defect assignments that never entered its TP text inventory: 10967-D03 in run a7f53199a759, and 14740-D01/14740-D05 in run 6de9765d8a56. They receive neither new TP nor advisory credit. These are separate from the uncredited classifier-bug counts above; preserving the published T/D accounting leaves this small historical undercredit unchanged.

Bug discovery is unchanged. Harnesses improve recall in 39/42 same-model/effort comparisons (mean Δrecall +0.135, Fable excluded for incomplete harness coverage). The peak-recall ratio remains 1.63× (0.599 versus 0.367). Legacy F1′, adjP′ and F2p_legacy retain the old non-bug-burden accounting for provenance; they are not alternative versions of the new advisory evaluator.

Full cell counts, sensitivity intervals and paired comparisons are in analysis/verified_gold/DEFECT_METRICS.json and analysis/final_report_metrics.json → true_gold_defects.verified.advisory_summary.

Figure 3.1a — What each cell found, and how much of it is noise interactive

How to read: goldens and hidden-gold defects count distinct verified bugs. Accepted useful advisories and unsupported findings count classified clusters. These are credited units, not a literal decomposition of raw emissions; advisories are not additional members of the bug universe. The components do not necessarily sum to raw reported findings. Missing advisory coverage is labelled unmeasured.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none opus glm-vis glm-flash sol fable sonnet astra terra

Harness all/none compound (CE) metareview (MRV) vanilla (one-shot)

Effort all/none medium high low

show 95% CIs

Static fallback (used automatically when JavaScript is unavailable)

What changed and what did not. The audited bug denominator remains 147 and the assignment repair is preserved. The new evaluator changes how useful non-bug findings are valued, so old F2′ rankings and ratios must not be carried forward. Recall, metered costs and token conclusions are unchanged. Of the highest-recall cell’s 59 misses, other complete configurations recover 52; the all-cell union remains 140/147. This is configuration complementarity, not measured repeat-run variance.

Per PR (§3.1).

Figure 3.1b — The quality frontier: how many bugs a cell finds versus how noisy it is (recall vs F2′) interactive

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none fable-5.1 opus sonnet glm-flash glm-vis sol terra astra

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

show 95% CIs

Static fallback (used automatically when JavaScript is unavailable)

What each cell is actually credited with.

Figure 3.1c — How many real bugs does each setup actually find — and how many do we miss? interactive

How to read: one horizontal bar per cell, showing the distinct real bugs it found out of the 147-bug true set, split into original goldens (the benchmark’s own 42) and hidden-gold verified defects (105 more); the marker is the cell’s coverage ceiling. Bars are ordered by framework within model.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none opus glm-vis glm-flash sol fable-5.1 sonnet astra terra

Harness all/none compound (CE) metareview (MRV) vanilla (one-shot)

Effort all/none medium high low

Static fallback (used automatically when JavaScript is unavailable)

Why no cell finds every verified bug. The denominator is not a list of things an agent could reasonably be expected to find; it is what the whole campaign (2,416 healthy runs, unioned and then audited) turned up. Three facts follow:

1. 1 of the 105 verified distinct defects was never reported by any run (10967-D05; surfaced by our own audit, real and test-validated, but no run’s findings describe it). It sits in every cell’s denominator and caps recall by a hair. Measured against only the defects some run actually reported (104), the denominators become 146 instead of 147 and recall moves less than half a point: the best-recall harness cell, opus · CE · medium, goes 0.599 → 0.603. This is a bug-denominator sensitivity check, separate from advisory-weight sensitivity. The reachable variant in DEFECT_METRICS.json reports this denominator. (Pre-audit this understated reachability more severely: 13 defects had zero credited findings, but the assignment pipeline had silently dropped 276 findings past a 60-finding cap and mis-assigned others — at least one of the “never reported” defects, 4-D56, had in fact been reported.)
2. A cell is one run per PR; the gold set is the union of the fleet. Averaged over the 66 complete cells, a cell finds 25.2 of 42 goldens (60%; best 35/42 = 83%) but only 20.7 of 105 hidden-gold defects (20%; best 53/105 = 50%). Official goldens are largely reachable; the hidden-gold tail is not.
3. The tail is heavy and the ceiling is ~95%. 30 defects have ≤3 assigned findings anywhere, 12 are found by exactly one of the 66 cells, and unioning all 66 cells still reaches only 140/147 (95%) — 6 defects (4-D38, 10-D20, 10-D21, 10-D22, 10967-D05, 11059-D33) and 1 golden are found by no complete cell at all. The defects no complete cell finds concentrate in PR 10 (3 of the 6).

Recall levels depend on the audited bug universe. F2′ additionally depends on the accepted advisory count, unsupported findings and the chosen α; it is not just a transformation of recall. Report all three counts together and use the α sensitivity alongside the point estimates.

Honest limitations. The adjudication/duplicate judges are non-deterministic (documented); duplicate calls used majority rules and fix-location evidence rather than a single judge’s word. The ground-truth construction, the 2026-09-18 evidence audit (3 withheld from the verified universe, 2 merged as duplicates) and the finding→defect assignment repair are described in §2.5, with per-defect evidence in WITHDRAWALS_AND_DEDUP_2026-09-18.md.

3.2 The benchmark’s own lens (its 42 golden comments), kept for comparison

66 of 72 cells are complete (6/6 PRs each; 396 runs, n = 1 run/PR, k = 1); the six fable compound/metareview cells have 1–2 PRs each (coverage gaps, not results — fable’s account was rate-capped during collection, and completing the grid was not funded; analysis/COVERAGE.md). xhigh is out of the matrix by operator decision (legacy rows aside: §3.5.4). Judges, batches, instruments per cell are in T1/T2 notes and analysis/COVERAGE.md.

T1 — apples-to-apples matrix (severity top-6): quality, 95% cluster-bootstrap CIs

recall = frozen-matcher TP/(TP+FN) vs the 42 goldens on these six PRs (All profile; Core/Strict differ only by PR 10967’s single style golden on this PR set). adjP = TP/(TP+hallucinations), k=1 verdicts; inst = precision instrument (rj3 = v3.1 clustered k=1, v2 = in-run three-way, v1 = binary legacy). n = PRs with a selected healthy scored run (1 run/PR). Judge: gpt-5.2 for fable/opus/sonnet/glm rows, claude-opus-4-5-20251101 for sol/terra/astra rows.

Reading the matrix. (a) Harnesses raise recall where the base model is weak or mid-tier and leave it unchanged or worse at the top of the frontier: +0.14…+0.31 recall for opus-5/glm rows (17/42 pairs resolve positive and 2 negative, i.e. 19 pairs whose paired CI excludes 0; median Δrecall +0.12), while sol/terra rows are inside noise (T4) — a strong model in a single pass already finds what the harness’s lenses find on these PRs. (b) sonnet-5 is the counterexample that matters for buyers: its compound cells lose recall (−0.24 [−0.46, −0.04] at high effort) because Claude-Code-style compound routing sends reviewer subagents to haiku-4.5 (visible in per_model_usage) — a harness is only as good as the models it actually calls. (c) adjP is where frontier harness cells bleed: opus-5 CE/MRV adjP 0.30–0.44 vs glm-vis MRV medium 0.85 and astra MRV low 0.93 — the expensive harnesses emit large volumes of non-golden content of which more is judged waste.

Figure 3.2a — Recall with 95% confidence intervals, per cell (strict 42-golden lens) static

How to read: one bar per cell, grouped by framework within model. Bar height is recall against the benchmark’s own 42 human-verified golden comments (the “strict lens”; our extended 147-bug ground truth is the primary lens in §3.1, and this panel is kept so the two can be compared). CI means confidence interval: the vertical whisker on each bar is the 95% CI obtained by resampling the six PRs 10,000 times (a PR-level cluster bootstrap — defined in full in §2.6). It summarizes sensitivity to resampling the observed PRs: wider intervals mean greater uncertainty under this resampling model. It is not a prediction interval for another six-PR benchmark and does not cover run-to-run model variation. Marginal whisker overlap is not a paired test; use the paired difference intervals to assess comparisons.

3.3 Framework comparison on the true golden set

MRV has a higher F2′ point estimate in 18 of 21 matched model·effort comparisons against CE; 8/21 differences favor MRV at 95%, 3/21 favor CE, and 10/21 include zero (§3.1). This is the primary framework comparison. The superseded semantic-union analyses and tables T13/T14 remain in Appendix A for provenance.

3.4 Cost, tokens, wall-clock, and the open-weight story

3.4.1 Price table (retrieved 2026-09-16)

Per 1M tokens, USD, standard tier, short-context rates. Metered $ = fresh_in·in + cached_in·cached + cache_w·w + (output+reasoning)·out, from per_model_usage per run. Our GLM traffic went through the Lunaroute gateway at a flat fee ($0 billed) — the metered figures are what the same token counts would cost on the public APIs. Reasoning output is billed at the output rate for OpenAI/GLM rows (it is a large share: e.g. glm-vis MRV high ≈ 470k reasoning tokens/run). Anthropic reconciliation in §2.3.4.

3.4.2 Cost & throughput per cell

T2 — cost & throughput per cell (top-6), 95% cluster-bootstrap CIs

Metered $ at published list prices retrieved 2026-09-16 (REPORT.md §3.4.1). tok/run includes fresh input + cached reads + cache writes + output (incl. reasoning). GLM cells marked * are conservative (all input priced at the fresh rate; no per-model token split available). Values < $0.01 are shown in cents (¢) to avoid leading-zero drowning; ≥ $0.01 in $.

Figure 3.4a — Cost per review, per cell (log scale) static

How to read: one point per cell; y is metered dollars per PR review at published list prices on a log scale, with 95% cluster-bootstrap whiskers; cells are ordered by framework within model.

3.4.3 Why one harness is cheap and another is not (token composition)

Figure 3.4b — Where a review's token bill actually comes from interactive

How to read: stacked token composition per review (fresh input, cached read, cache write, output) for low-effort cells; use the full dashboard’s effort selector for medium and high. The cached-read share is what makes a frontier harness’s blended per-token rate look cheap. Hover any bar for its component volumes.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none sonnet opus astra sol terra glm-vis glm-flash fable-5.1

Harness all/none metareview (MRV) compound (CE) vanilla (one-shot)

Effort all/none low

Static fallback (used automatically when JavaScript is unavailable)

The frontier harnesses are input-cache machines — opus-5 CE/MRV burn 1.7–4.4M tokens per PR, ~75–90% of them cached reads at 10% of list input ($0.50/Mtok), which is why their blended rate drops to 0.14–0.18 ¢ per k-token ($0.0014–$0.0018). The GLM harnesses burn fewer total tokens (about 100–744k/PR across complete GLM harness cells) at list input rates ($1.40 fresh / $0.26 cached for vision). Both arrive at cheap per token; they differ by ~13× in $ per PR review at the recommendation point (vision-MRV $0.22 vs opus-CE $2.95) and by ~13× in tokens per task against the opus harness cells (§3.5.2). The vanilla cells are trivially cheap per task because a single call is ~100k tokens — but they find the fewest real issues (T4 recall deltas).

3.4.4 Wall-clock

Figure 3.4c — Does a slower setup buy a better review? interactive

How to read: wall-clock seconds per PR review (x) against F2′ (y), one point per cell; points further right took longer per review. Toggling the 95% CI checkbox shows the cluster-bootstrap whiskers. The same seconds appear in the wall column of T2.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none fable-5.1 astra sol opus-5 glm-vis terra sonnet-5 glm-flash

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

show 95% CIs

Static fallback (used automatically when JavaScript is unavailable)

At low effort the recommendation cells are latency-competitive with everything: glm-vis MRV low 95 s/run [77, 114] vs opus-5 CE low 110 s [99, 121] and opus-5 MRV low 100 s [91, 108]. At medium/high effort the GLM lane is throughput-limited by the gateway (glm-vis MRV medium 2,282 s [1,458, 3,305]; glm-flash MRV high 2,984 s) — an operational, not quality, difference (§3.5.5).

3.4.5 The efficiency frontier (headline figure)

Both figures below use the true golden set (§3.1: 42 goldens + 105 verified hidden defects with archived reproduction/fix-test evidence = 147) and both use F2′ — our evaluator (verified bugs plus accepted useful advisories, with unsupported findings penalized) for the quality axis. Axis limits include the full confidence intervals. A strict-benchmark version of each figure remains in git history (data freeze 2026-09-16).

Figure 3.4d — The efficiency frontier: cost vs recall, and cost vs F2′ interactive

How to read: every complete cell plotted as metered dollars per real finding (log x) against recall (left panel) and F2′ (right panel); use the CI checkbox to show uncertainty. The frontier is the upper-left envelope of point estimates. Numbered keys identify its cells; axes include all CI endpoints.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none glm-flash glm-vis sonnet terra sol opus fable-5.1 astra

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

show 95% CIs

Static fallback (used automatically when JavaScript is unavailable)

The frontier x-axis uses metered dollars per real finding under the original campaign adjudication, including repeated findings; it is not recomputed from the new deduplicated A/T counts. The y-axis uses the new F2′. This x-axis differs from dollars per distinct verified bug in Figure 3.4e. Read the generated plot for its current envelope rather than treating a frontier as a production guarantee. Recall retains the premium opus · CE · medium endpoint, with 88/147 bugs found. F2′ also rewards accepted advisory breadth, so the two panels answer different questions. Cells missing advisory coverage are not eligible for the comparable F2′ frontier.

Figure 3.4e — What it costs to catch one true bug interactive

How to read: the x-axis is the price per distinct true bug — one of the 147 in our golden set — and the y-axis is F2′, a composite that rewards verified bugs and accepted useful advisories while penalizing unsupported findings. Its weights are declared preferences, not measured developer costs. Cheap-and-high is the win: a point far to the left but low is buying cheap catches while missing a lot; far right and high is thorough but you pay for it. One point per complete cell, coloured by model; hover for the value and its CI, double-click a legend entry to isolate a model.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none fable-5.1 astra sol opus-5 glm-vis terra sonnet-5 glm-flash

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

show 95% CIs

Static fallback (used automatically when JavaScript is unavailable)

Interactive versions of all main figures — single-file HTML, no server needed: analysis/figures/interactive_dashboard.html (open in any browser; built by tools/final_report_html.py). A single global filter key area (checkboxes for all 8 models, 3 frameworks, 3 efforts) drives every panel at once. The primary quality metrics use the audited true-gold set; benchmark-golden selection checks and labelled historical views remain separate. These are benchmark measurements, not production estimates. Sections contain independent panels, each with a question-as-title, how-to-read note, own CI toggle (off by default), own chart, takeaway callout and click-details card — (1a) price/performance ($/run vs F2′); (1b) latency/quality (wall vs F2′); (1c) cost per true bug ($/distinct verified bug vs F2′); (1d) cost per original-campaign real finding ($/real vs F2′); (1e) token volume (tokens/run vs F2′); (1f) stacked verified bugs found (goldens and hidden defects); (2) effort ladder per framework; (3) token composition per effort; (4) sample-bias check — a dumbbell chart of the pairs (full-50 gray vs top-6 colored, sorted by gap, recall/F1 toggle); (5) per-PR selection view (severity vs recall, the six chosen PRs in bold color). Style follows the OWID/NYT minimal-chrome school: direct annotation of the recommendation cell, muted palette, no chartjunk, all uncertainty visible.

3.5 The five claims, tested

3.5.1 Harnesses spend more tokens but find more bugs — both numbers, always

The claim, and why both numbers must travel together. A harness finds more than a one-shot prompt because it makes the model work harder — more passes, more tool calls, more tokens. The honest way to report that is to quote both sides at once, because a quality gain at any price is not something a buyer can act on. Table T4 does exactly that for all 42 matched harness-vs-vanilla pairs on the same PRs, and the pattern is consistent: a harness typically spends 10–20× the tokens and costs roughly 3–5× the dollars of the same model run one-shot, while raising recall on those PRs. Two things make that trade worth reading carefully rather than dismissing on the multiple alone. First, the multiple is measured per review, not per bug: a harness that costs 4× per review but finds 12× the real bugs is strictly cheaper per bug found (that comparison is the subject of §3.4 and Figure 1). Second, the multiple is a property of the model’s price, not of the harness design — which is why running a harness on an open-weight model costs less than a frontier model run once, and that inversion is the report’s cost headline.

Across 42 harness-vs-vanilla pairs on the same 6 PRs.

Table T4 — harness vs vanilla, paired on the same 6 PRs (Δrecall CI; token ×; cost ×) — full table in Appendix B, Table T4.

Summary (42-golden benchmark lens): median token multiple 10.1× (van about 3.5k–270k tokens → harness about 100k–6.4M), median Δrecall +0.12; resolved positive (CI excludes 0) in 17/42 pairs, negative in 2 (sonnet CE high and terra CE high), unresolved in 23. Cost multiples 0.6×–20×. The pairs where the harness pays for itself in recall are opus-5 (CE low +0.14 [0.02, 0.25] at 20× tokens), the GLM rows (vision MRV low +0.26 [0.12, 0.41] at 9.6× tokens, 8.0× cost; flash MRV low +0.31 [0.18, 0.49] at 9.4× tokens, 8.1× cost) and astra (MRV low +0.21 [0.10, 0.33] at 13.6× tokens). sol/terra pairs: token multiples with recall gains inside noise — a frontier single pass is already near these harnesses’ ceilings on hard PRs.

3.5.2 Per-token price vs per-task cost — the operator’s “~1/200th” is not supported

The operator’s shorthand was “~1/200th per token, ~1/10th per task”. Measured at published list prices with the token mixes these runs actually used, neither ratio holds. The direction of the folklore is right — the open-weight lane is dramatically cheaper — but its magnitude is not: against the matching frontier one-shot run the cheap GLM-flash harness is about 1/57 per blended token and about 1/38 per task. The two ratios differ because a harness task spends more tokens than a one-shot task does (§3.5.1), so the per-task advantage is smaller than the per-token advantage — quote whichever you mean, and check which one a vendor is quoting.

The per-pair ratios are below (collapsed so they do not interrupt the argument):

Key rows:

• glm-flash MRV low vs fable-5.1 vanilla low: per-token 1/57 (0.017 [0.016, 0.019] of fable’s blended rate), tokens/task 1.5×, net cost/task 1/38 (0.026 [0.024, 0.028]).
• glm-flash MRV low vs opus-5 vanilla low: per-token 1/26, tokens/task 1.5×, cost/task 1/17.
• glm-vis MRV low vs opus-5 vanilla low: per-token 0.35 (NOT 1/200 — opus vanilla is a high-volume single call at list rates), tokens/task 1.6×, cost/task 0.55.
• Against a commercial harness reference: glm-vis MRV low vs opus-5 MRV medium: the per-token ratio is about 0.980, the token-volume ratio 0.0462, and the per-task cost ratio 0.0453. Both rates and volume enter the cost identity.

The correct summary is not “1/200th per token” but: the flash tier is ~1/25–1/60 of frontier per blended token and ~1/17–1/40 per task; the vision tier is ~1/3 of frontier per token vs opus vanilla; vision MRV-low is also cheaper per task than the low-effort vanilla references (about 25% of Fable, 55% of Opus and 54% of Astra). Token overhead partly (not fully) offsets the cheap rate — that is why both ratios must be quoted together.

3.5.3 Higher thinking budgets do not reliably buy more bugs — a null result, stated as one

Under the 42-golden benchmark lens, there are 22 high-vs-medium paired comparisons.

Table T5 — effort ladder: high vs medium, paired (ΔF1 CI; Δrecall CI; cost high/medium) — full table in Appendix B, Table T5.

Summary: 17 not resolved by this sample; 4 resolved positive (opus-5 vanilla ΔF1 +0.24 [0.08, 0.39] at 1.30× cost; sonnet-5 vanilla +0.09 [0.01, 0.18] at 1.54×; glm-flash CE +0.09 [0.01, 0.20] at 2.44×; glm-flash MRV +0.12 [0.03, 0.18] at 4.17×); 1 resolved negative (astra CE ΔF1 −0.09 [−0.21, −0.01] at 1.16× cost — high effort made it worse). Cost multiples for unresolved pairs are 0.96×–3.0×. A buyer pays 0.96×–4.2× for high vs medium and in 17/22 model×framework cases cannot measure a quality difference on 6 PRs. For the GLM rows the cost multiple is the steepest (reasoning tokens at output price) with the least demonstrated gain.

3.5.4 Too little capability collapses recall — the floor exists, and where it starts

Within the matrix, flash vs vision.

Table T6 — glm-flash vs glm-vision, paired on the same 6 PRs (Δ from vision → flash) — full table in Appendix B, Table T6.

Summary (42-golden benchmark lens): paired golden-recall differences are unresolved at low/high effort (Δrecall CIs include 0 in 8/9 cells; the exception is MRV medium, vision +0.14 [0.02, 0.28]) — but beyond-gold real findings drop sharply for flash: −39 [−62, −16] (CE low), −104 [−179, −45] (CE medium), −201 [−266, −141] (MRV medium) per cell (6 PRs). Flash finds the goldens’ core but surfaces far less of the surrounding real-issue space; its adjP is also lower (more noise). Below the matrix, the legacy rows show the collapse outright: glm-5.2- vision-flex vanilla on the top-6 recalls 0.28 (medium, n=5) / 0.45 (xhigh, n=6) and ~3 real/PR vs glm-vis vanilla’s 0.55–0.62 and 9.5–12.7/PR; kimi-k3 and gpt-5.2 have 1–2 healthy top-6 runs (unusable as cells). The capability floor sits between glm-5.2 and glm-5.3; flash has unresolved golden-recall differences in most comparisons and lower breadth where that drop is demonstrated (CE low/medium and MRV medium; at MRV low the delta is +3 [−56, +64] — no demonstrated drop).

3.5.5 The recommendation (supported by this sample, with its stop-holding conditions)

The two low-effort MRV configurations remain useful pilot candidates: vision and the substantially cheaper flash. Their current common-instrument comparison is:

Vision costs about 9.4× as much and finds one additional distinct bug, with 14 fewer scored unsupported findings; flash receives 10 more accepted advisories. Accepted advisories and unsupported findings are separate outcomes, rather than one combined noise count. Its F2′ interval is 0.567 [0.490, 0.673], compared with flash’s 0.556 [0.462, 0.712]. Marginal intervals do not resolve a paired difference or establish equivalence. This experiment does not establish which saves more developer time; measure triage time and usefulness on your own reviews.

Table T8 — recommendation ratios (glm @ low vs frontier reference cells, same 6 PRs) — full table in Appendix B, Table T8.

42-golden benchmark comparison (not true-gold metrics): recall 1.00× [0.91, 1.09] of opus-5 CE low (0.81 vs 0.81), F1 1.28× [1.00, 1.58] (0.73 vs 0.57), at 7.5% [6.7, 8.5] of its metered cost ($0.22 vs $2.95/run; $0.039 vs $0.52 per golden TP; $0.0037 vs $0.048 per real finding) and comparable wall-clock (95 s vs 110 s). vs opus-5 CE high: recall 1.13×, F1 1.65×, at 3.2% of cost. vs fable-5.1 vanilla low: recall 1.10× at 24.6% of cost. glm-flash MRV low is the budget option: recall 1.03× of opus CE low at 0.8% of its cost, adjP 0.54 (more noise to triage).

Conditions under which this stops holding (measured, not speculative): (1) Gateway throughput — at medium/high effort the GLM lane ran 4–12× slower than the frontier cells at the same effort (up to ~30× against the fastest frontier low-effort cell) (§3.4.4) and the 2026-09-14/15 incident showed the cheap lane’s capacity limits; the recommendation is a low-effort recommendation. (2) Pre-fix infrastructure confound — the headline GLM cells predate the gateway/SDK fixes (§2.3.4); quality direction unknown, likely operational. (3) Breadth — if beyond-gold real findings matter (they do for review value), the demonstrated flash breadth losses occur at CE low/medium and MRV medium; MRV-low shows no resolved breadth loss (§3.5.4). (4) The top-6 sample — hard-PR-weighted; §2.4.2 checks only benchmark-golden recall and precision across the observed full-set cells; it does not validate true-gold generalization (top-6 overstates harness F1 by ~+0.08; the glm-vis MRV low full-50 F1 is 0.57 vs top-6 0.73 — the ranking survives, the level does not). (5) Single judge family per row and k=1 adjudication (§2.3.1, §4).

3.6 Does the six-PR choice distort benchmark-golden metrics? (selection effect and sampling)

Table T3 — selection effect: top-6 vs full-50 (cells with ≥40/50 healthy scored PRs) — full table in Appendix B, Table T3.

Rank agreement (Spearman of model F1 ranks, top-6 vs full-50, within framework×effort):

Full sampling disclosure: analysis/COVERAGE.md.

Summary: 33 cells with ≥40/50 PRs back the top-6 with full-50 runs; benchmark-golden recall has a small observed mean gap (mean gap +0.006), harness F1 is overstated by +0.084 on average, model rankings mostly agree (Spearman 0.80 median; CE-low 0.37 the exception). The fable compound/metareview cells remain gaps (budget decision after collection limits), opus/sonnet vanilla are top-6-only by operator decision, GLM partial fills were stopped at the data freeze.

Figure 3.6a — Does the six-PR sample distort the picture? interactive

How to read: one point per cell, showing that cell’s recall on the selected six highest-severity PRs minus its recall on the full 50-PR benchmark (Δ, x-axis; the vertical line is zero). A point right of zero did better on the selected six than on the full set — the opposite of a “these PRs were harder than average” reading. Models are grouped, so each key entry is one model’s ladder of cells. (The dashboard adds an F1 toggle for this panel; the embedded version shows recall.)

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none fable-5.1 astra sol opus-5 glm-vis terra sonnet-5 glm-flash

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

Static fallback (used automatically when JavaScript is unavailable)

Figure 3.6b — Can you trust the six PRs this report went deep on? interactive

How to read: one point per PR in the benchmark — x is the summed original-golden severity weight (Critical = 4 … Low = 1), y is the per-PR benchmark-golden recall. The six PRs this report went deep on are highlighted (with their mean), against the other covered PRs. This embedded panel shows Astra · vanilla · high; use the dashboard’s cell selector to inspect other configurations.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none astra

Harness all/none vanilla (one-shot)

Effort all/none high

Static fallback (used automatically when JavaScript is unavailable)

3.7 Effort ladder and cost curves

Figure 3.7a — The effort ladder: does more thinking buy more quality? interactive

How to read: metered dollars per run (x, log scale) against F2′ (y), one point per cell, with each model’s low→medium→high effort steps connected inside its framework. Movement up-and-right is buying quality with money; movement sideways (or down) is not.

Key

Filter this chart: untick a model, harness or effort to drop its points.

Models all/none fable-5.1 astra sol opus-5 glm-vis terra sonnet-5 glm-flash

Harness all/none vanilla (one-shot) compound (CE) metareview (MRV)

Effort all/none low medium high

Static fallback (used automatically when JavaScript is unavailable)

4. What we cannot claim

1. We did not run Fable 5.1 across all PRs and all harnesses — our choice, for budget. Fable 5.1 was run one-shot on the full primary sample, but its compound-engineering and metareview cells only on 1–2 PRs each: a complete Fable 5.1 harness grid (all PRs × all frameworks) was more than this campaign could fund, and we spent that money on depth instead. The partial Fable harness numbers in T1/T2 are gaps, not results, and no claim in this report rests on them. We will amend the report with updated data if a sponsor funds the runs, or if someone runs them per §2 and sends us the data — see §2.2’s caveats.
2. n = 1 run per cell×PR: cluster bootstrap quantifies PR-sampling variance only; run-to-run LLM variance is not in these CIs (tools/eval_adjudicator.py, analysis/score_flips.py outputs, and analysis/ADJUDICATOR_INTERRATER.md measure judge-side variance only). Some vanilla cells have duplicate runs (coverage doc) that we did not pool.
3. Single judge family per row; cross-judge sensitivity only partially measured. gpt-5.2’s ~55–61% run-to-run self-agreement and the 0.80 intra-5.6-generation agreement cluster are documented in analysis/ADJUDICATOR_INTERRATER.md (the latter is shared-error convergence, not correctness).
4. k=1 adjudication (campaign lock): v3.1 clustering removes phrasing flips, but single-vote residual noise remains; §2.3.3’s v1-vs-rj3 deltas illustrate instrument risk; they do not bound it or measure vote noise.
5. Provider incident (§2.3.4): all GLM top-6 harness cells contain pre-fix runs.
6. Matched-excluded / profiles: all headline numbers are All-profile (see §2.1); Core/Strict differ by ≤1 golden on this PR set.
7. Cost is metered list price, not our invoice (gateway flat fee); retries are inside per_model_usage for in-run calls, but gateway-level failed-call overhead (~5.4k failed GLM calls in the ledger, tools/key_usage_report.py) is excluded from per-run figures and is an operational surcharge on the cheap lane during the incident window.
8. Advisory usefulness is model-adjudicated. A common GLM-base k=1 classifier applies the existing staff-review bar; this is not human validation. Same-family judging may favor GLM outputs, and confidence scores do not measure calibration. The reference advisory set covers only two of six PRs, so advisory recall is unmeasured. α sensitivity tests a declared preference, not uncertainty in labels. More accepted advisory volume can raise F2′ without finding another bug; read T/A/H and recall together.

5. Reproducibility

Three distinct modes — do not conflate them:

5.1 Setup

Tested environment (2026-09-18): Python 3.14.7, numpy 2.5.2, matplotlib 3.11.1 — pinned exactly in requirements.txt. Numeric regeneration is deterministic and offline after dependencies are installed. HTML rendering additionally uses Markdown 3.10.3 and Plotly 7.1.0; static dashboard snapshots use Kaleido 1.4.0 and an installed Chrome/Chromium. The dashboard-fragment exporter uses Node.js (tested: 22.23.2). Image bytes can depend on the browser, fonts and rendering platform.

git clone https://github.com/dsifry/harnesseval.git && cd harnesseval
# Use the source revision containing this corrected report and its checksum manifest.
# The older report-2026-09-18 tag is the pre-correction edition.
git rev-parse HEAD
git status --short              # retain any local changes as part of the source snapshot
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt     # numeric analysis + HTML/chart rendering dependencies

Source snapshot. The historical report-2026-09-18 tag resolves to 8f7b8f9c, before the calculation, provenance and presentation corrections in this edition. Use the source checkout that contains this corrected Markdown and its matching OUTPUT_SHA256SUMS; the old tag alone will not reproduce these corrected artifacts. Until the corrections are committed, retain the working-tree changes as well as the base commit. The correction and validation record is analysis/REPORT_CORRECTIONS_2026-09-18.md.

To re-run an eval yourself (not needed to reproduce the published numbers, which are stored), see INSTALL.md §3.4–§3.5: install the harness deps (pip install -e .), provide the pinned third_party/ checkout (or authenticated gh for PR diffs), then run one cell with run_model_matrix --fill. Verified end to end: vanilla-engineered / glm-5.3-flash-background / low on PR 8 ran in 42 s, produced 10 findings, was judged by gpt-5.2, adjudicated (4 real / 1 hallucination), scored TP=5 FP=6 FN=1, and registered in runs/.

API keys: none are needed to recompute statistics from saved judgments. The numeric chain below is deterministic and offline. Only the model-running steps (which produced the runs/ registry and the stored LLM artifacts) need keys: follow INSTALL.md §2.3 and use your own OpenAI / Anthropic credentials. Lunarroute is optional — it is merely the OpenAI-compatible gateway this campaign happened to use for the open-weight GLM/Kimi lanes; any router or direct provider works (HARNESS_LUNAROUTE_* in harnesseval/keys.py). No key material is committed: the harness reads keys from a file outside the repo (~/.config/harnesseval/keys.env, chmod 600).

Original campaign inputs (the pre-correction chain was verified in a fresh clone; see §5.3):

The benchmark’s golden comments are vendored at analysis/inputs/golden_comments/ because the upstream checkout lives in third_party/, which is gitignored (and is a nested git checkout, so its files cannot be committed from here). Provenance and SHA-256 checksums are in that directory’s README.md. The tools prefer the vendored copy and fall back to third_party/ if present; if neither is found they now abort with a FATAL message rather than writing a dataset with empty denominators.

5.2 The chain (run in this order)

# 1. ingest the run registry (needs only the committed runs/ + the golden comments)
.venv/bin/python tools/final_report_extract.py            # runs/*        → analysis/final_report_dataset.json

# 2. the §3.1 per-cell metrics, then the registry's derived tier counts/table
.venv/bin/python tools/verified_gold_defect_metrics.py    # dataset+registry → analysis/verified_gold/DEFECT_METRICS.json
.venv/bin/python tools/verified_gold_registry_sync.py     # defect meta.json → DEFECT_REGISTRY.{json,md} (_final counts)

# 3. the headline metrics, including the §3.1 true-gold section it appends
.venv/bin/python tools/final_report_compute.py            # dataset + DEFECT_METRICS → analysis/final_report_metrics.json
                                                          #   (frozen §1–§3.7 + the Appendix A unions + true_gold_defects)

# 4. the catalogue, then the presentation layer
.venv/bin/python tools/gold_defect_catalog.py             # registry → GOLD_DEFECT_CATALOG.{json,md,csv}
.venv/bin/python tools/final_report_figures.py            # → analysis/figures/*.png + interactive_dashboard.html
.venv/bin/python tools/final_report_figures_true_gold.py  # → the true-gold charts
.venv/bin/python tools/final_report_tables.py             # → markdown tables (writes /tmp/final_report_tables.md)
.venv/bin/python tools/final_report_html.py               # → analysis/figures/interactive_dashboard.html

# 5. rebuild the current report's interactive charts and static fallbacks
.venv/bin/python tools/report_interactive_charts.py      # independent chart fragments + manifest
node tools/export_dashboard_panels.js --ci              # dashboard fragments, including CI variants
.venv/bin/python tools/dashboard_panel_snapshots.py      # requires installed Chrome/Chromium
.venv/bin/python tools/report_to_html.py                 # REPORT.md + EXECUTIVE_SUMMARY.md → matching HTML
.venv/bin/python tools/sdlc_report_html.py               # metrics + dataset → sdlc-report.html (the developer edition; stdlib only)

# 6. the two scope/ledger checks the report cites (no keys needed)
.venv/bin/python tools/verify_hitlist.py --verbose        # top-6 scope check (§2.4). NOTE: exits 1 while
                                                          #   hitlist rows remain unrun - expected, 29/111,
                                                          #   the disclosed fable coverage gap. Informational.
.venv/bin/python tools/key_usage_report.py                # GLM gateway ledger (§4 caveat)

Inputs vs outputs. These files are committed inputs produced by the verification campaign, not by this chain, and must be present: runs/, analysis/inputs/golden_comments/, analysis/exp_union_semantic_pilot_*.json, analysis/semantic_*.json, analysis/verified_gold/DEFECT_REGISTRY.json, analysis/verified_gold/DEFECT_ASSIGN.json and the per-defect evidence directories. The chain writes the rest. Step 1 and step 3 each abort with a FATAL message if a required input is missing, rather than writing a degenerate artifact.

Common advisory instrument inputs. The separate model-judgment archive is analysis/verified_gold/advisory_readjudication/: the selected pass manifest records configuration and hashes; active_pass.json identifies the active archive rather than making old passes current. prompts.json and diffs/ retain prompts and full cached PR diffs. The clusters/, verdicts/ and runs/ directories retain cluster plans, judgments and the frozen per-run adjudication records the offline chain consumes; progress.json records current-pass progress. These preserve decisions and provenance. Raw per-call payloads (pair_calls/, pair_attempts/, group_calls/, grouping_calls/, grouping_attempts/, scoring-policy attempts/) are excluded from the repository by .gitignore — they are mode-(iii) artifacts reproducible only by new model calls — with one deliberate exception: the hash-bound policy audit and the frozen execution manifest pin the sha256 of every grouping and pair call they consumed, and those ~800 checkpoint payloads are tracked so validation can run offline (the mode-(iii) entry point tools/advisory_timeout.py verifies the manifest before making any call). Original run judgments and verified bug assignments are preserved. These saved judgments are inputs to offline metric regeneration, not outputs an offline command can invent. The raw pass is passes/glm_base_claim_v3/; the selected policy is scoring_policies/advisory070_penalty080_v1/, selected by active_scoring_policy.json. Its policy.json and completed per-PR groups/ artifacts freeze the ≥0.70 advisory identities. tools/advisory_scoring_policy.py validates source hashes and applies A ≥0.70/H ≥0.80 without changing raw judgments. Missing or stale policy grouping is not accepted as a valid scored result. tools/advisory_policy_dedup.py contains the frozen grouping implementation; the current model-calling entry point is .venv/bin/python -m tools.advisory_timeout, which applies the archived 120-second timeout extension. This belongs to mode (iii), not the offline numerical chain. tools/advisory_framework_comparison.py filters the same identities at ≥0.80 and writes the matched framework_comparison.json and FRAMEWORK_COMPARISON.md into the policy directory. To refresh current true-gold statistics and the confidence-cutoff comparison from saved judgments:

.venv/bin/python tools/verified_gold_defect_metrics.py
.venv/bin/python tools/final_report_true_gold.py
.venv/bin/python -m tools.advisory_framework_comparison

Re-running tools/readjudicate_report_advisories.py is mode (iii): it makes new model calls, requires provider access, and need not reproduce the same labels. Its manifest records the actual judging configuration. The common-pass prompt adds the explicit staff advisory criteria while preserving historical rj3 defaults. The selected report policy credits A ≥0.70 and penalizes H ≥0.80, inclusive. Reused semantic evidence, grouping decisions and raw responses are archived separately, making the changed instrument explicit.

The from-scratch call path was verified in a fresh clone with independent key configuration (2026-09-19): tools/pilot_report_advisories.py — the bounded, isolated trial — completed all 16 fresh calls (7 classification controls + 1 grouping, repeated at concurrency 4 and 6) against the user’s own gateway credentials; every classification verdict gated into the F2′ categories (bug / hallucination / important_non_bug / unresolved) and the grouping mapping passed validate_group_mapping. Repeated calls reproduced identical gated verdicts for only 4/8 requests — the measured, expected nondeterminism that is exactly why these steps are mode (iii).

5.3 What reproduces exactly, and what does not (measured, not asserted)

Expected output checksums (sha256; verify with shasum -a 256 -c analysis/verified_gold/OUTPUT_SHA256SUMS from the matching source snapshot — the manifest covers the artifacts below and confirms your checkout carries the published artifacts exactly. Given the pinned inputs and requirements, the §5.2 numeric chain deterministically regenerates its metric artifacts; DEFECT_ASSIGN.json is a stored mode-(iii) artifact the chain consumes, not one it regenerates. Figures may still differ in bytes across matplotlib versions, which the pins narrow):

Corrected-edition verification (2026-09-19). This edition was re-verified the same way: fresh clone, every chain output deleted, §5.2 re-run in order. All five manifest artifacts matched OUTPUT_SHA256SUMS, and every regenerated figure (PNG and SVG), interactive fragment, dashboard, table and HTML rendering was byte-identical to the committed artifacts (a clean git status). SVGs previously embedded matplotlib’s creation date; the figure tools now suppress it (§5.4 item 4), which is what makes the SVG twins and the inlined-SVG REPORT.html byte-reproducible. The test also caught two evidence gaps — the frozen per-run adjudication records (passes/*/runs/) and the audit-referenced checkpoint payloads — which are now tracked (see §5.2).

For the pre-correction edition, the chain was re-run in a fresh clone with every chain output deleted first (so a match cannot be a stale file matching itself) and compared byte-for-byte. Full record, including the eight problems the test found and fixed: analysis/REPLICATION_TEST_2026-09-18.md. The historical results below describe that reproduction test; the corrected edition’s checks and changed artifact hashes are recorded separately in analysis/REPORT_CORRECTIONS_2026-09-18.md.

5.4 Tool changes made for this report (disclosed for transparency)

1. tools/scoreboard.py — added an Fb column with --beta (default 2.0, the benchmark’s recall-weighted default). F1/recall/adjP semantics untouched: adds ap.add_argument("--beta", ...) and one line per row fb = (1+b*b)*rec*adjp/max(1e-9, b*b*adjp+rec).
2. New analysis tools (no frozen instrument touched): tools/final_report_extract.py, tools/final_report_compute.py, tools/final_report_figures.py, tools/final_report_tables.py, tools/final_report_true_gold.py, tools/verified_gold_defect_metrics.py, tools/gold_defect_catalog.py, tools/final_report_figures_true_gold.py, and the verified_gold_*.py verification/merge tools.
3. Input resolvers + loud failure: the eight tools that read the golden comments now prefer analysis/inputs/golden_comments/; final_report_extract.py and final_report_compute.py abort instead of writing a degenerate dataset when the golden set is empty.
4. tools/final_report_figures.py and tools/final_report_figures_true_gold.py — two determinism fixes for the SVG twins: every SVG savefig call now passes metadata={'Date': None} (omitting matplotlib’s <dc:date> creation timestamp), and svg.hashsalt is pinned to a fixed string (matplotlib otherwise derives SVG element IDs from a per-session random salt). Chart data, layout and all other SVG content are unchanged; same- environment regeneration is now byte-identical.
5. tools/report_to_html.py — the rendered HTML no longer embeds the generation date (datetime.date.today()), which made REPORT.html/EXECUTIVE_SUMMARY.html differ on every calendar day. The provenance line now reads “Generated by tools/report_to_html.py”; rendered content is otherwise unchanged.

harnesseval/judge.py, readjudicate3.py semantics, and the golden dataset were not modified.

6. Further work

Funding and sufficient provider capacity to complete the Fable harness grid; opus/sonnet vanilla full-50 fill; rj3 k≥3 re-adjudication as a sensitivity layer; cross-judge recall matching; anchor-level correctness via tools/anchor_matcher.py; human-preference study on important_non_bug volume; line-anchored correctness analysis (not yet reported); resuming the GLM fill lanes (--skip-batch health-gated).

Appendix A. Superseded analyses (retained for provenance)

Historical definitions: scores and interpretations below retain their original metric definitions. Any historical F2′ charged important non-bugs as burden; it is not the advisory-reward F2′ now defined in §2.6. These historical tables must not be mixed into current F2′ rankings.

Superseded. The material in this appendix is retained for provenance only. The primary results are in §3.1; nothing in this appendix should be cited without that context.

A.1 The superseded key-union analysis (the earlier clustering approach)

Appendix A introduced the semantic union and was then audited twice more. Both audits found real defects, and both are corrected here:

1. Under-merge. A stricter whole-PR LLM re-merge (same judge, all cluster representatives in one call instead of file-chunked) collapsed 359 → 258 clusters: PR 11059 alone went 83 → 35. The 359 was an overcount of distinct defects.
2. Golden overlap. 47 clusters describe defects already in the golden set — the official text-only matcher missed them, so the adjudicator filed them as real_but_ungold. Counting them as additional inflated both the denominator and the per-run credit. Judged at the finding level (a cluster overlaps a golden only if a single member finding is that golden’s exact defect), then low-confidence cases re-judged strictly; 47 survive and are excluded from the additional set.

Verified set: 42 goldens + 211 additional real bugs = 253 distinct bugs. Every additional bug carries a human-verifiable card (location, why-real quoting the code, replication steps, and the cells that found it) in analysis/TRUE_GOLDEN_EVIDENCE.md; the 47 golden-duplicates are listed there as removed. Judge gpt-5.2, 2026-09-17; artifacts are the record. Total LLM cost of the verification chain ≈ $30.

Metrics. recall_sem = (goldens found + additional bugs found) / 253. Two credit rules: A (benchmark-compatible) counts only goldens the official matcher credited; B (PRIMARY, real-world) counts goldens found officially ∪ goldens whose verified overlapping cluster the run produced — a set union, so no bug is ever counted twice. adjP charges only hallucinations (claim soundness); adjP′ also charges nitpicks (the user lens). F1/F1′ pair recall_sem with adjP/adjP′. CIs: cluster bootstrap, B=10,000, seed 20260916, rng5=SEED+4. This revision fixes the paired bootstrap metric-axis indexing and makes harness-versus-vanilla draws pool counts just as their point estimators do. Historical paired intervals change; primary true-gold metrics do not. Cells with one covered PR have a degenerate (zero-width) CI — flagged in T12.

Headline (T12).

What the verification changed. - Recall levels rise (~0.44–0.46 for the best cells vs 0.33–0.41 under Appendix A) because the denominator shrank and golden-FN credit was restored — but the ordering and the comparison verdicts do not move. - MRV still beats CE: mean ΔF1 +0.044 [+0.019, +0.073] (17+/4− over 21 paired model·effort comparisons); ΔF1′ +0.042 [+0.024, +0.063]. - Harness-vs-vanilla holds: 39/43 paired Δrecall_sem resolve positive on this superseded semantic-union metric (primary true-gold count: 39/42). - Rule A vs B differ by <0.01 on harness cells (e.g. glm-vis·MRV·high 0.435 vs 0.443); they matter more for weak cells.

Why even the best harness misses so much — three mechanisms, now separated. 1. Denominator inflation (fixed here): paraphrase splits and golden duplicates made the set look ~1.6× bigger than the final universe (253 after the strict re-merge; 147 after the 2026-09-18 audit — 3 withdrawals + 2 duplicate merges; the raw pre-merge count was 359); a large part of the apparent miss was counting the same bug repeatedly. 2. Configuration complementarity, not blindness and not measured run-to-run variance: using the pre-merge cluster graph, the best cell found 133 of 359 clusters but 91% of what it missed was found by another harness configuration; collectively harness cells found 94% of clusters. Different models/efforts/frameworks surface different slices; with one run per cell per PR, this experiment does not measure what a re-run of the same setup would recover. 3. A genuine blind tail of 12 (listed in TRUE_GOLDEN_EVIDENCE.md): verified bugs no harness cell found are almost all a different class — performance/N+1/eager-loading, test-quality gaps, framework idioms (String#<< mutation, Handlebars empty-array truthiness, \z anchors), migration-lock semantics. The lenses hunt correctness/security; these are outside their brief. A scope explanation was tested and rejected: among anchored findings, 227 are in the PR’s changed files vs 1 out-of-diff.

Table T12 — verified-union matrix (Appendix A; superseded by the §3.1 primary matrix): per-cell metrics with 95% cluster-bootstrap CIs — full table in Appendix B, Table T12.

Table T15 — verified key-union per PR (Appendix A; superseded by the §3.1 per-PR table) — full table in Appendix B, Table T15.

Total: 42 goldens + 211 verified additional bugs = 253 distinct bugs (258 merged clusters; 47 golden-duplicates removed, not counted). Per-bug verification cards (location, why-real, replication, found-by): analysis/TRUE_GOLDEN_EVIDENCE.md.

A.2 Expanded-gold union tables (superseded by §3.1)

SUPERSEDED by §3.1 (levels only; the Appendix A unions were intermediate steps). The T9/T10/T11 unions below overcount distinct bugs ~17× (paraphrase splits) and, due to a since-fixed extract bug, omit all rj3-adjudicated runs’ confirmed bugs (706 findings, mostly vanilla cells). The paired Δ directions survive; all levels should be read from §3.1. Retained for provenance.

The strict-benchmark numbers above answer the question Martian defines: does the tool find the human-verified golden comments, without hallucinating? They deliberately do not credit findings outside the golden set — which is the right call for a benchmark, but the wrong lens for a buyer comparing a single-pass reviewer against a harness: a harness that finds 55 real bugs but only 8 goldens scores the same recall as a vanilla that finds 8 goldens, and the 47 real-but-ungold bugs are invisible to the metric. This section separates the two analyses and recomputes recall/precision/F1 against an expanded ground truth.

Construction (our extension, fully disclosed). Per PR, take every confirmed-bug finding (rj3 bug / in-run real_but_ungold) across ALL healthy scored runs of all models/frameworks/efforts (era-legal universe, low/medium/high). Cluster them with a file:startline:endline primary key extracted from each finding’s own text prefix (the tools/anchor_matcher.py pattern; 45% of bugtexts carry an anchor), with rj3-normalization + difflib 0.75 (bucketed by file path) as the fallback for anchorless findings. The expanded set = goldens ∪ distinct keys (T11). Per cell: tp_exp = golden TP + own distinct keys; fn_exp = expanded size − tp_exp; adjP_exp = tp_exp/(tp_exp + hallucinations); F1/F2_exp from the pair. Cluster bootstrap CIs (B=10,000, rng3 — frozen numbers above untouched). important_non_bug is excluded (bugs only). Caveats: no LLM semantic-merge pass, so cross-model rewordings stay separate and the union is overcounted ⇒ recall_exp levels are conservative lower bounds; distinct issues sharing one anchor can over-merge; golden-vs-cluster overlaps may double-count a few entries. The robust quantity is the paired Δ (T10), which is stable across all three clustering variants we tried (verbatim, difflib-only, anchor-primary).

Table T11 — hidden-gold union sizes per top-6 PR — full table in Appendix B, Table T11.

Table T9 — expanded-gold matrix (top-6): recall/adjP/F1 against the hidden-gold union — full table in Appendix B, Table T9.

Table T10 — harness vs vanilla, paired on the same 6 PRs: Δrecall under the expanded set — full table in Appendix B, Table T10.

Resolved positive 38/42, negative 0, unresolved 4 (strict-benchmark version: 17/42 positive, 2 negative — see T4).

Historical reading (superseded). These expanded-union numbers preceded the audited true-gold analysis in §3.1 and are retained only for provenance. Under the expanded set, the harness-vs-vanilla comparison flips decisively: 38/42 paired Δrecall_exp resolve positive (0 negative; 4 unresolved), vs 17/42 positive / 2 negative under the strict benchmark (T4). The vanilla cells collapse (recall_exp 0.003–0.011 — they find the goldens but almost none of the hidden-gold space), while the harness cells keep meaningful coverage (0.03–0.10). The cells whose strict-benchmark harness-vs-vanilla deltas were unresolved (sol/terra) now resolve positive; sonnet-5’s compound cells — the strict benchmark’s one resolved negative — move to unresolved/marginal. The recommendation cell (glm-vis · MRV · low) holds and strengthens: recall_exp 0.054 [0.045, 0.064] — 9× fable-5.1 vanilla low (0.006) and comparable to opus-5 harness cells, at the same fraction of the cost. Levels are conservative; the ranking and the ratios are the reportable quantities.

A.3 Superseded semantic framework comparisons

Table T13 — CE vs MRV, paired on the same PRs (Δ = MRV − CE, semantic metrics) — full table in Appendix A, Table T13.

Pairs-level aggregate: mean ΔF1 +0.044 [+0.019, +0.073], mean ΔF1’ +0.042 [+0.024, +0.063] — resolves positive. Signs: ΔF1 17+/4−/00; ΔF1’ 18+/3−. Δrecall: 6 positive, 2 negative, 13 unresolved paired intervals (95%). ΔF1: 7 positive, 2 negative, 12 unresolved paired intervals (95%). ΔF1′: 9 positive, 0 negative, 12 unresolved paired intervals (95%).

T14 — harness vs vanilla, paired Δrecall_sem (semantic union)

With the corrected pooled-count bootstrap: 39 positive, 0 negative and 4 unresolved intervals on this superseded semantic-union metric. The primary true-gold result is 39/42 positive point estimates; these are different denominators and different claims.

Appendix B. Data tables

Every supporting data table is collected here, collapsed by default and linked from the text where it is discussed. They carry the same numbers as the machine-readable artifacts (analysis/final_report_metrics.json, analysis/verified_gold/DEFECT_METRICS.json): the narrative sections state each finding, and these tables are the full evidence behind it. Three tables stay in the section that discusses them, also collapsed, because they are the primary evidence for that section’s own claim — T1 (§3.2, the benchmark’s own lens), T2 (§3.4, cost and throughput) and T14 (§3.3, harness vs vanilla on the semantic union).

About this revision

(corrected report revision 2026-09-18 · based on pre-correction tag report-2026-09-18; see §5)

Campaign: 20260910-mrv0120-manifold (+ vanilla reuse batches, era rule in §2.3.2) · Data freeze: 2026-09-16 09:35 (verified: no data-generating lanes alive at 09:45; every number below is computed from runs/*/summary.json as of the freeze — re-run the commands in §5 and you will get these numbers) · This report supersedes archive/report-2026-08-26-batch083.md, archive/report2-2026-08-26-batch083.md, docs/FRAMEWORK_COMPARISON.md.

Companion documents: EXECUTIVE_SUMMARY.md (1–2 pages, decision-first), analysis/COVERAGE.md (coverage table + sampling disclosure), figures in analysis/figures/ (embedded in the HTML rendering), machine-readable numbers in analysis/final_report_metrics.json / analysis/final_report_dataset.json.

Note on naming: this file was REPORT_FINAL.md; the FINAL name was dropped because the report is a living revision, not a final artifact. The historical August-2026 reports live in archive/.

Disclosure: the author maintains metareview (MRV), one of the harnesses evaluated in this report. It is open source, MIT licensed, and available free of charge at https://github.com/dsifry/metareview.

Special thanks

Thank you to the team at LunaRoute, who supported this work with an account and free usage while their service was in alpha. The open-weight GLM review lanes in this report ran through their gateway (§3.4.1), and that help was much appreciated. The Anthropic and OpenAI usage in this campaign was paid for by the author.

LunaRoute serves open-weight models (GLM-5.3, GLM-5.3-Flash, DeepSeek-4.1-flash and many others) from US datacenters on flat-rate plans, and its stated policy is that request and response bodies are processed in memory and never written to disk. In the author’s experience it was fast and low-latency.

Disclosure: the author has no relationship with LunaRoute other than as a happy user, and used the service free of charge as part of the alpha.

This page is a generated HTML rendering of REPORT.md — the Markdown file is the source of truth. Generated by tools/report_to_html.py from the checked-out Markdown and figure artifacts. Figures are regenerated renderings; the interactive charts carry the same data as the interactive dashboard.