A lightning talk · September 2026

Vibes vs. Evidence

Harnesses find more verified bugs, and open-weight models make them cheap. A systematic evaluation of AI code review: prompts, models, effort levels, and harnesses. Measured, not guessed.

Dave Sifry · github.com/dsifry/harnesseval
→ next ← back F fullscreen O overview
The problem

Everyone will tell you what works.

Just use the biggest model. It’s obviously better.
Crank the thinking budget to max.
Agent harnesses are token-burning theater.
Open-weight models can’t do serious code review.
I heard open models cost 1/100th as much as Fable.
Works great for me.
VIBES
Confident. Contradictory. Unmeasured.
So we ran the experiment

8 models × 3 ways of working × 3 effort levels.

Same pull requests. Same judges. One-shot prompt vs. two agentic review harnesses: compound engineering (parallel persona subagents) and metareview (deterministic gates + adversarial LLM lenses).

2,416healthy review runs across the campaign
147real bugs in the answer key, on the six highest-severity PRs of a public 50-PR benchmark
Every $metered: tokens, cache, wall-clock, list prices
66 of 72 cells complete. The six Claude Fable 5.1 harness cells (hollow) ran on only 1–2 PRs: unfunded, so treated as gaps, not findings.
Finding 1

The benchmark was hiding most of the bugs.

42
bugs the public benchmark’s human reviewers labelled in these six PRs.
real bugs. The 105 new ones each own an executed test: it fails on the PR, and passes on the fix.
of 147 is the most any single configuration finds. All 66 combined find 140. Seven, nobody finds.
True golden set = 42 benchmark goldens + 105 individually test-verified hidden defects. A post-publication audit withdrew 3 and merged 2 duplicates; all numbers here are post-audit. Best cell: opus-5 · CE · medium.
Finding 2

Same model. Add a harness. More real bugs.

18 of 21
matched model × effort comparisons: metareview’s F2′ point estimate exceeds compound engineering’s. Paired 95% intervals favor MRV in 8, CE in 3; 10 include zero.
Marks = mean real bugs across three effort levels. Bug discovery still improves in 39/42 harness-vs-one-shot comparisons (+0.135 mean recall). On the matched 21-cell cohort, mean advisory F2′: MRV 0.444 · CE 0.399 · one-shot 0.265. A ≥ 0.70; unsupported H ≥ 0.80.
The catch

Harnesses burn ten times the tokens.

So it must cost a fortune… right?
Median tokens per PR review across the 42 matched pairs. Median harness ÷ one-shot token multiple: 10.1× (typically 10–20×).
Finding 3 · the money chart

What does a dollar per review buy?

one-shot prompt harness · commercial model harness · open-weight GLM ● one-shot   ■ compound (CE)   ▲ metareview (MRV)  ·  hover any point
F2′ point estimates on the 147-bug true set: verified bugs + useful advisories (A ≥ 0.70), unsupported claims penalized (H ≥ 0.80) · six PRs · 66 complete cells · prices retrieved 2026-09-16. Similar point estimates are not statistical equivalence.
BUT…
Next vibe

So what’s the optimal effort level?

low? Medium? High? xhigh? MAX???
ULTRA
We ran low, medium and high on every model and framework. xhigh and beyond were outside this study’s matrix.
Finding 4

More thinking does not reliably buy more bugs.

17 of 22
comparisons: we found no statistically significant difference between high and medium effort. Four favored high; one favored medium.
Lines: each model × framework’s F2′ (true set) from low → medium → high effort. 17 of 22: paired high-vs-medium ΔF1 under the benchmark-golden lens, 95% CI includes zero. High effort cost 0.96×–4.2× as much as medium. Six PRs; no clear difference does not prove equal performance.
Finding 5

Even the pricing folklore was vibes.

The vibe · “I heard…”
1/100th
→
In our comparison
1/57 per token

…and 1/38 per review, including input, output, and caching costs—even though the harness uses about 1.5× as many tokens.

“I heard…” is illustrative hearsay, not an attributed quote. Measured: GLM Flash + MRV low vs. Fable one-shot low on the same six PRs, using actual token mixes and list prices from 2026-09-16. Per-review cost includes input, output, cache reads and writes where charged.
Science means saying this part out loud

What we can’t claim.

Six PRs, chosen for severity.

Two codebases, the hardest slice of a 50-PR benchmark. A deliberate stress test, not a random sample of your repo.

One run per configuration per PR.

Our intervals cover which PRs we sampled. Run-to-run model variance isn’t in them.

Best-vs-best is selected after seeing results.

Opus · CE · medium leads at F2′ 0.641; best eligible one-shot is Fable · medium at 0.427. Their paired difference is +0.214 [0.116, 0.275], but selection makes it optimistic.

The Fable 5.1 harness grid is unfunded.

Frontier model × many-pass harness was beyond budget. Those cells are gaps. Run them, send the data, and we’ll amend the report.

Read the intervals, not the rankings. Run your own evals on your own workloads.
What you can do now (or tell your CTO)

A strong review pilot at $0.22 per PR.

The recipe

GLM-5.3 · metareview · low effort

$0.22per PR review
95 swall-clock
72/147real bugs found
0.567F2′
Budget option

glm-5.3-flash · metareview · low

$0.024per PR review
71/147real bugs · more noise to triage
vs. leading cell — opus-5 · CE · medium: $6.12 · 343 s · 88/147 · F2′ 0.641  (27.6× the recipe’s price)
Lower review cost makes broader coverage worth testing.
An open-weight, strong-but-not-frontier model inside a harness. A low-effort recipe: medium/high-effort GLM cells took up to ~3,000 s/review; the cause is not isolated here. Measure triage time and developer benefit in a pilot.
One thing to remember

Don’t trust the vibes.
Run the eval.

github.com/dsifry/harnesseval
Harness > one-shotMore real bugs in 39 of 42 matched comparisons.
MRV usually leads CEHigher F2′ point estimate in 18 of 21 matched cells.
Max thinking ≠ more bugs17 of 22 comparisons: no statistically significant difference between high and medium effort.
scan for the repo
Dave Sifry · david@sifry.com · full report, data, and reproduction chain in the repo
Author maintains metareview, one of the harnesses evaluated here · open source, MIT licensed · github.com/dsifry/metareview