harnesseval · September 2026 · the developer edition

Vibes vs. Evidence: What actually delivers AI code review quality

By Dave Sifry (GitHub · LinkedIn). This is the short version for working developers. The technical report has the methods, confidence intervals, limitations, and steps to reproduce the results.

We measured which AI code review setups find real bugs, what they cost, and how long they take. Here is what to use and why.

What we measured. Eight models each reviewed the same pull requests three ways (as a single engineered prompt, and running the open-source harnesses Compound Engineering and metareview) at three effort levels. Every review was scored against 147 verified bugs on the six highest-severity pull requests, inside a campaign of 2,416 review runs across a 50-PR public benchmark. How we kept score ↓

Disclosure. I wrote metareview (MRV), one of the harnesses tested here. It is open source (MIT) and free. Every harness was judged by the same frozen instruments, and the data and code to re-run the analysis are public. Full disclosure and safeguards →
Don’t take our word for it. Pit any model, harness and effort level against each other across all 66 complete configurations: bugs found, noise, dollars and seconds.
Compare them yourself
Finding · harness

A harness finds more real bugs than a single prompt

A single prompt (“one-shot”) gives the model the diff and a carefully written review prompt, one time. Read the exact prompt (REPORT §2.2). A harness makes the same model work the review in passes, with tools and specialised reviewers. We tested the open-source harnesses Compound Engineering (“CE” in the charts) and metareview (“MRV”). With the same model and effort level, the harness found more verified bugs in 39 of 42 comparisons, a mean gain of 13.5 percentage points of recall—the share of verified bugs a review found.

Verified bugs found, one-shot versus harness

Each row is a model; marks show the mean across the three effort levels, out of 147 verified bugs. Fable is omitted because its harness runs were not completed.

Every model moves right. The best harness configuration found 88 bugs; the best one-shot configuration found 54. The three losing comparisons were all Sonnet 5 running Compound Engineering.
Show this chart as a table

Verified bugs found by AI code review, one-shot prompt vs harness, same model. The harness won 39 of 42 comparisons; best harness 88 bugs vs best one-shot 54 (of 147). Data and method:

What the extra findings cost

A harness spent a median of 10× the tokens, about 5× the dollars and 3.5× the time per review of the same model run one-shot. At low effort the median harness review took 2.8 minutes, against 54 seconds for one-shot.

There is also more to check: the median harness configuration produced 5.5 unsupported findings (“hallucinations”) across the six reviews against 0 for one-shot, alongside 27.5 accepted useful suggestions against 6.5. More useful findings, but more to check.

Results and uncertainty for each comparison → REPORT §3.5.1

Finding · cost

The best value: an open-weight model running a harness

Choose an open-weight model to keep the cost of a harness down. A harness uses more tokens, so the model’s token prices matter. The measured comparison: GLM-5.3 running metareview at low effort scored 0.567 at $0.22 a review and found 72 bugs. Opus 5 running the same harness at the same effort scored 0.567 at $2.97 and found 74. That is 1/13 the cost, with the same score at the precision shown. Scores and uncertainty → REPORT §3.1

Why Opus 5 and GPT-5.6 Sol? They were the strongest Anthropic and OpenAI models when running a harness, and each is priced a tier below its vendor’s flagship (price table → REPORT §3.4.1). GPT-5.6 Sol peaked at 0.535 (GPT-5.6 Sol running Compound Engineering at high effort, $0.73 a review), a lower score than the open-weight pick at a higher cost. Of the 30 closed-model harness configurations we completed, only Opus 5 running Compound Engineering at medium effort outscored it, and none was cheaper. All 12 open-weight harness configurations outscored every closed-model one-shot prompt (0.478 or better, against a best of 0.427). Four of the top five scores overall are open-weight models.

What a dollar per review buys

Every complete configuration. Further left is cheaper; higher is better. The line is the efficiency frontier: the best score available at each price. Faded points cost more and score lower than a point on the line. The score (F2′) rewards verified bugs most, credits useful advice and penalises unsupported claims.

Highest absolute score is in the upper right. The best value is in the upper left. Opus 5 running Compound Engineering at medium effort scored highest with no adjustment for price: 88 bugs and 0.641, at $6.12 a review. GLM-5.3-Flash running metareview at low effort found 71 bugs for $0.024. It is the cheapest configuration to find 70 or more, at roughly 1/260 of the price. GLM-5.3 running metareview at low effort, a different open-weight model, found 72 for $0.22 with fewer unsupported findings (5 against 19). The line marks the best score available at each price: every configuration on it is an open-weight model except the most expensive (Opus 5).

What a dollar per AI code review buys. Top score: Opus 5 · Compound Engineering · medium, 88 verified bugs at $6.12. Tightest budget: GLM-5.3-Flash · metareview · low, 71 at $0.024. Open-weight models own the value end:

Side by side: quality, price and time

Finding counts are totals across six pull requests; cost and time are averages per review.

The score is built from the bug, suggestion and unsupported columns. Verified bugs count most. A useful suggestion (an “accepted advisory” in the full report) is a finding that is true and worth a maintainer’s attention but is not a correctness bug or a style nit; it earns credit. An unsupported finding is a false claim, a style-only comment or vague speculation; it costs points. Bugs are proven by tests. Suggestions and unsupported findings are judged by a model, and a finding the judge was unsure about counts as neither. How the score works → REPORT §2.6

Per token and per review are different costs

Per token, GLM-5.3-Flash running metareview cost 1.7% of what Fable 5.1 cost as a one-shot prompt. Per review it cost 2.6%, because a harness review uses more tokens than a single prompt. Both are real; they are not the same number.

Use low effort for fast, inexpensive reviews

Within each model and harness, low effort was the fastest setting in 14 of 14 cases and the cheapest per verified bug in 10 of 14. Running metareview at low effort, GLM-5.3 took 95 seconds a review, GLM-5.3-Flash took 79 seconds and Opus 5 took 100 seconds.

Turn the effort up and the open-weight models get slow. GLM-5.3 running metareview took about 38 minutes a review at medium effort and 49 minutes at high, against 3.1 minutes and 4.0 minutes for Opus 5 running the same harness. If a review has to come back while the author still has the change in mind, stay at low effort.

Prices, token use and review time → REPORT §3.4

Finding · effort

More effort mostly made the bill bigger

We compared high and medium effort for the same model and the same way of running it, on the same pull requests. When a confidence interval includes zero, this sample cannot tell high and medium apart. That happened in 17 of 22 comparisons. Across all comparisons, high effort cost 0.96× to 4.2× as much.

How to read the bars

Each bar is a confidence interval: a range of effects consistent with these results. The range comes from how much the result changed from one pull request to the next. A dot to the right of zero is not a clear win if its bar extends across zero. For example, Sonnet 5 running metareview shows a gain of +0.09, but its interval goes from −0.07 to +0.26. High effort may have helped, done nothing, or hurt; this sample cannot tell which. Each configuration ran once, so the bars do not include variation from running the same setup again. That is an additional source of uncertainty.

Quality change from medium to high effort

Each row is a model and a way of running it. The dot is the change in F1, a score that balances bugs found against false alarms. This comparison uses the original 42 benchmark labels, rather than the expanded set of 147 verified bugs. The bar is the confidence interval. The cost multiple is on the right. Effort comparison and limitations → REPORT §3.5.3

Grey rows: no statistically significant quality difference. High effort significantly improved F1 in four comparisons and significantly reduced it in one.
Show this chart as a table

Does “high” reasoning effort find more bugs in AI code review? In 17 of 22 paired comparisons the gain over medium was not measurable, at 0.96× to 4.2× the cost. Chart and data:

High effort cost more in 20 of 22 comparisons and significantly improved F1 in four. The largest F1 gain was Opus 5 as a one-shot prompt (+0.24, interval +0.08 to +0.39). Even there, a harness at low effort found more verified bugs: Opus 5 running either harness found 65 and 74, compared with 47 for the same model using a single prompt at high effort.

Unsupported findings fell more often than they rose. We did not formally test high against low effort. In raw counts, high effort found an average of 5 more bugs than low, out of 147. Unsupported findings went down in 13 of 22 configurations and went up in 4.

More pull requests and repeated runs could clarify the effects this sample cannot distinguish. The recommendation stays the same: use a harness, at low or medium effort. Make high effort earn its extra cost on your own workloads. Open questions → REPORT §6

The full effort ladder → REPORT §3.5.3

How we kept score

We only counted bugs we could prove

The public benchmark’s human reviewers labelled 42 bugs in these pull requests. The models kept reporting more. So we audited them: every one of the 105 additional bugs has a test that fails on the pull request and passes on the fix. That gives 147 verified bugs to score against.

42bugs the benchmark labelled
147verified bugs after our audit
88the most any single configuration found
140found by all 66 configurations combined; seven were found by none

Why six pull requests, when the campaign ran 50

The full campaign made 2,416 healthy review runs over all 50 benchmark pull requests, in five codebases. It is expensive to prove every bug with a test. So we ran the complete grid only on the six highest-severity pull requests: every model, with every harness, at every effort level. We chose these pull requests with a rule that we set before the test. Every comparison on this page uses this set.

Then we compared these pull requests with the full benchmark, using the benchmark’s own labels across the 33 configurations that also ran all 50 pull requests.

What held up: each configuration found about the same share of the labelled bugs on the six as on all 50 (a mean gap of +0.006), and the model rankings stayed the same or nearly the same in four of five groupings.

What did not: harnesses looked better on the six than they do across the whole benchmark. On the other, lower-severity pull requests they raised more unsupported findings, which pulled their overall quality score (F1, which balances bugs caught against false alarms) down by about 0.08 on a zero-to-one scale. GLM-5.3 running metareview at low effort, for example, scored 0.73 on the six and 0.57 on all 50.

So read this page for which setup beats which, not for the exact scores, and expect a harness to be somewhat noisier on lower-severity pull requests than these numbers suggest. The full check → REPORT §2.4.2

How the ground truth was built and audited → REPORT §2.5  ·  The full-benchmark check → REPORT §3.6

Your turn

See for yourself: compare any model, harness and effort level

All 66 complete configurations, on the axes you care about. Filter, hover for the numbers, and click any point or table row to pin it for a side-by-side comparison. The link to your comparison is shareable.

Pinned for comparison

Leaderboard

Follows your filters. Click a column to sort, a row to pin.

This is not a contest between the harnesses. Read it for harness against one-shot, open-weight against closed, and what each costs: the top 19 configurations are all harness runs, and the dot shows how each review was run.

Scores describe these six pull requests, with a single run per configuration. Close scores are not a ranking: intervals and sensitivity checks are in REPORT §3.1.

Compare eight models × three ways of working × three effort levels for AI code review: verified bugs found, noise, dollars and seconds per review. Interactive, with all the data:

What to do with this

Next steps for you and your team

  1. Run a harness, not a single prompt.If your AI review is a single prompt over the diff, this is the biggest lever we measured: more verified bugs in 39 of 42 like-for-like comparisons.
  2. Use an open-weight model at low effort for value.Both harnesses are open source and free. Compare them on your own pull requests. Running metareview, GLM-5.3 at low effort found 72 bugs at $0.22 and about 95 seconds a review; GLM-5.3-Flash is the budget option at $0.024 (71 bugs), with more unsupported findings to check (19 against 5). Running Compound Engineering, GLM-5.3-Flash at low effort was the cheapest and fastest harness configuration we measured: 63 bugs at $0.019 in 49 seconds, with 11 unsupported findings. Compound Engineering also holds the top score overall (Opus 5 at medium effort: 88 bugs at $6.12 a review).
  3. Vet the provider before you send it your code.A review sends your source code to whoever hosts the model, and hosts’ terms differ widely. Work through the provider checklist ↓
  4. Default to low or medium effort.Make high effort earn its place on your own pull requests before you pay for it everywhere.
  5. Measure the time you spend checking findings.We measured bugs found, noise, dollars and seconds. We did not measure developer time saved, and that is the number that decides whether any of this is worth it for your team.

The recommendation and the conditions under which it stops holding → REPORT §3.5.5

Before you send it your code

A checklist when choosing an open-weight model provider

Open weights mean many companies can host the same model, and you can host it yourself. The model is the same; the terms are not. A code review sends your source code to whoever runs the inference, so ask these questions of any host before a pilot. This is separate from the recommendation above: it applies whichever open-weight model and harness you choose.

Pass it on

Know a team choosing an AI code reviewer? Hand them the evidence

Share a finding with your team or a wider audience. Each post starts with the takeaway, then gives the evidence, limits, and source links. Copy it into LinkedIn or a team channel, edit it in your own voice, and attach the matching chart.

Find more bugs with the model you already use

Your AI code reviewer may not need a smarter model. It may need a better workflow. The same models found 1.6× as many verified bugs when a harness ran the review. That was the median gain from adding a harness across 42 comparisons in an AI code review study. A harness runs the model through several review passes, with tools and specialist reviewers, instead of asking it to review the code in a single prompt. → More verified bugs in 39 of 42 comparisons with the same model and effort level → Best harness setup: 88 bugs found. Best single-prompt setup: 54 → GLM-5.3 with metareview at low effort found 2.1× as many bugs as its single-prompt setup The gain has a price: median token use was about 10× higher, cost 5× higher, and review time 3.5× longer. There were also more unsupported findings to check. A higher bug count does not automatically mean less work for a developer. Use a harness. Then measure how much time its findings save on your own code. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#f-harness Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html What would change if your reviewer found more bugs with the model you already use?

A comparable score for a fraction of the cost

Why pay $2.97 for an AI code review when $0.22 matched its quality score? GLM-5.3 matched Opus 5 using the same harness, at 1/13 the cost. GLM-5.3 running metareview at low effort matched Opus 5 running the same harness and effort in this study. The score rewards verified bugs and useful advice, and penalises unsupported findings. → GLM-5.3: $0.22 per review, 72 bugs found across the test set → Opus 5: $2.97 per review, 74 bugs found → Both rounded to the same quality score: 0.567 The cheaper model is open-weight: its weights are available for others to host, subject to its license. The result is a reason to choose an open-weight setup for value, not a claim that all models perform equally. Speed matters too. The open-weight setup took 95 seconds per review at low effort, but 38 minutes at medium and 49 minutes at high. Keep effort low for a fast review. Check the host’s data policies before sending code. These are list-price estimates. Close scores do not establish equal performance on other workloads; run your own evals. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#f-cost Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html What would your team do with that difference in review cost?

Make high effort earn its bigger bill

Do you set your AI to High, Max or Ultra and wonder why the results aren’t satisfactory? In 17 of 22 AI code review comparisons, the study could not establish a quality difference between high and medium effort. High cost more in 20 of 22. → High effort significantly improved the quality score in four comparisons → It significantly reduced the score in one → Costs ranged from 0.96× to 4.2× the medium-effort price The score here is F1: it balances bugs found against false alarms. This effort comparison used the original 42 benchmark labels, rather than the expanded verified-bug set. An uncertain result does not mean the settings are identical. It means this sample cannot tell them apart. Default to low or medium effort. Make high effort earn its extra cost on your own code. High versus low was not formally tested, and each setup ran only once, so this is not proof that extra reasoning never helps. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#f-effort Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html Has your team measured what high effort actually buys?

See what changes when the same model gets a harness

The model gets all the attention. The review workflow deserves more. Harnesses beat single prompts in 39 of 42 comparisons—with the same model and effort. This chart compares a single review prompt with a harness: several review passes, tools, and specialist reviewers. Each row keeps the model the same. The marks average its results across effort levels. → The harness found more verified bugs in 39 of 42 same-model, same-effort comparisons → The average gain was 13.5 percentage points in recall—the share of verified bugs found → The best harness setup found 88 of 147 bugs; the best single-prompt setup found 54 It was not a win in every comparison. The three losses were all Sonnet 5 running Compound Engineering. Harnesses also used more tokens and produced more unsupported findings to check. The single prompt was carefully written, with a reviewer role, bug categories, severity levels, and instructions to report only confident findings. Its exact text is in the technical report. The practical takeaway: the way a model reviews code deserves as much attention as the model you choose. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#c-harness Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html Have you compared a harness with your current review prompt?

Find the review setup that fits your budget

AI code review for $0.024 a pull request sounds too cheap to be useful. That setup found 71 of 147 verified bugs across the test set. Here’s what paying more actually bought. That was GLM-5.3-Flash running metareview at low effort in this AI code review study. The bug count is a total across the selected pull requests; the price is the average per review. This chart puts quality against price: → Highest score: Opus 5 · Compound Engineering · medium, 88 bugs, $6.12 per review → Lowest cost to find at least 70 bugs: GLM-5.3-Flash · metareview · low, $0.024 per review → Fewer unsupported findings than that budget pick: GLM-5.3 · metareview · low, $0.22 per review (5 against 19) Left is cheaper. Up is a higher score. The line connects the best score available at each price. All but its most expensive point are open-weight models, whose weights are available for others to host under their licenses. The score rewards bugs and useful advice, and penalises unsupported findings. The leading scores have overlapping uncertainty ranges. Use the chart to choose a setup, then measure its results and the time spent checking findings on your own code. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#c-money Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html Which matters most for your team: more bugs found, fewer false alarms, or a lower bill?

See where high effort helped—and where it did not

“Think harder” is easy advice when someone else pays the bill. High effort cost more in 20 of 22 comparisons, but significantly improved review quality in just four. This AI code review chart compares high and medium effort for the same model and review method. It shows the change in F1, a score that balances bugs found against false alarms, using the original 42 benchmark labels. → Green: a statistically significant improvement → Red: a statistically significant decline → Grey: the sample cannot establish a difference The bar shows uncertainty across pull requests. If it includes zero, high effort may have helped, done nothing, or hurt. Grey does not mean “no effect,” and it does not tell you the cost. Cost is shown separately on the right. High effort cost more in 20 of 22 comparisons, with ratios ranging from 0.96× to 4.2× across all comparisons. Each setup ran once; the bars do not include variation from repeating the same run. Default to low or medium effort. Use your own evaluation to decide when a higher bill buys a result worth paying for. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#c-effort Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html What evidence would make you turn high effort on?

Check your own model choices against the data

Is your AI code reviewer worth what you pay? Compare 66 setups side by side—bugs caught, unsupported claims, price, and speed. See where your setup stands. This interactive explorer lets you compare models, review methods, and effort levels on the same pull requests. A harness runs several review passes with tools and specialist reviewers; the alternative is a single review prompt. → Filter to the models you care about → Compare bugs found, quality scores, useful advice, and unsupported findings → Put those results against dollars, seconds, or tokens per review → Pin setups side by side and share the comparison The study’s main findings: harnesses found more bugs in 39 of 42 comparisons; GLM-5.3 running metareview matched Opus 5 at 1/13 the cost; high effort often cost more without a clear improvement over medium. Every number comes from the same data as the technical report. Bug counts are totals across the test set; costs and times are averages per review. Small score differences are not reliable rankings, and the study did not measure developer time saved. Choose a setup from the evidence, then run your own evals on your own workloads. The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public. Finding and chart: https://dsifry.github.io/harnesseval/#c-explore Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html Where does your current reviewer land?

There is much more behind this page

What the full report adds

Read this before you quote us

What this does not prove

The short version

The findings, each one a decision you can make

“Verified” means proven: 42 bugs labelled by the benchmark’s human reviewers plus 105 more, each with a test that fails on the pull request and passes on the fix. How we kept score ↑

Finding · harness
1.6×
Same model, add a harness: 1.6× verified bugs found by AI code review

A harness runs the model as a team of specialist reviewers instead of a single prompt. Compound Engineering and metareview beat the same model’s one-shot prompt in 39 of 42 comparisons. GLM-5.3 running metareview at low effort gained 2.1×.

The price: about 10× the tokens, and more unsupported findings to check.

AI code review: same model, add a harness (Compound Engineering or metareview) and it found 1.6× the verified bugs, beating one-shot prompting in 39 of 42 same-model comparisons at about 10× the tokens. Data and method:

Finding · cost
1/13the cost
Best value: open-weight GLM-5.3 running a harness matched Opus 5

GLM-5.3 running metareview at low effort scored 0.567 at $0.22 a review. Opus 5 running the same harness scored 0.567 at $2.97. GPT-5.6 Sol peaked at 0.535, at $0.73 a review.

The single best score is still Opus 5 running Compound Engineering at medium effort (0.641), at $6.12 a review. Intervals overlap: read “comparable”, not “better”. Fable’s harness runs were not completed.

AI code review: open-weight GLM-5.3 running metareview at low effort matched Opus 5 in the same harness (0.567 vs 0.567) at 1/13 the cost ($0.22 vs $2.97 per PR) and outscored GPT-5.6 Sol:

Finding · effort
17of 22
Default to low or medium: high effort rarely bought a measurable gain

In 17 of 22 head-to-heads, high effort showed no statistically significant quality gain over medium, at 0.96× to 4.2× the cost. It helped in four and hurt in one.

Measured on the benchmark’s original 42 labelled bugs. High versus low was not formally tested.

Paying for “high” reasoning effort in AI code review: in 17 of 22 head-to-heads it showed no measurable quality gain over medium, at 0.96× to 4.2× the cost. Helped in four, hurt in one. Data:

Special thanks

Acknowledgments

Thank you to the team at LunaRoute, who supported this work with an account and free usage while their service was in alpha. The open-weight GLM reviews on this page ran through their gateway, and that help was much appreciated. §3.4.1 I paid for the Anthropic and OpenAI usage myself.

Disclosure. I have no relationship with LunaRoute other than as a user, and I used the service free of charge as part of the alpha.

Thank you to Ian Langworth (@statico on X and GitHub) for reading through the copy and suggesting improvements to the wording and clarity.