LLM Code Reviewer Reliability, by the Numbers

How reliable is an LLM when you point it at a diff and ask what’s broken? The measured answer: strong at finding bugs, weak at saying the same thing twice. A 2025 benchmark clocked the recall of three frontier models between 0.78 and 0.88 on a set of real security defects, well above what the deterministic scanners in the same test managed. The catch is what those models couldn’t do — pin a finding to the right line, or return the same verdict when the prompt ran again.

That pairing is the whole story. An LLM reviewer has real recall and a reliability bill that comes due the moment you try to make it a gate. A gate has one job: mean the same thing on every run. The numbers below come from three studies, and they argue for a division of labor, not a winner.

What the Benchmark Measured

BrassCoders, the scanner that catches what AI assistants structurally miss, runs on a deterministic-plus-LLM split, which is why the study worth reading first is arXiv 2508.04448: it put three LLMs against three deterministic analyzers across 10 real C# projects holding 63 known vulnerabilities.

The three models — GPT-4.1, Mistral Large, DeepSeek V3 — ran against three deterministic analyzers: SonarQube, CodeQL, and SnykCode. Full paper: Large Language Models Versus Static Code Analysis Tools by Damian Gnieciak and Tomasz Szandala, with the models run through the GitHub Models platform.

The defects span SQL injection, cross-site scripting, hardcoded secrets, and command injection. Two caveats ride along: the code is C#, and the deterministic tools tested aren’t the ones BrassCoders bundles. What carries across languages is the shape of the result — which class of tool is strong where, and where each one breaks.

Recall: Where an LLM Reviewer Wins

BrassCoders hands its findings to an AI assistant for a reason the recall numbers make plain: in the benchmark, the three models scored recall between 0.78 and 0.88, while the deterministic scanners came in between 0.19 and 0.52. On raw catch rate, the models won.

Recall is the fraction of real defects a tool actually flags. The paper credits the models’ lead to it directly: their F1 scores ran roughly 0.75 to 0.80 against 0.26 to 0.55 for the static tools, and the authors attribute that gap to the models’ ability to reason across broader code context. A rule fires only when the pattern it encodes matches. A model can flag a defect it has never seen a signature for. That’s the recall you want somewhere in your pipeline.

Precision and False Positives: The Bill for That Recall

BrassCoders treats an AI reviewer’s output as triage rather than verdict, and the precision numbers show why: the three models scored 0.72 to 0.78 precision, so roughly one flag in four was a false positive, and DeepSeek V3 posted the highest false-positive ratio of the group.

Precision is the fraction of a tool’s flags that are real. At 0.75, a quarter of what the model reports is noise a developer has to clear. The fair reading of this benchmark is narrow: the deterministic tools scored lower precision here (0.57 to 0.69), so this isn’t a case of static analysis being cleaner. The point is about consistency, because the false-positive rate swings by model, and the paper notes DeepSeek’s rate raises the verification effort a developer has to spend. For a check that runs on every commit, a flag rate that’s a quarter wrong and model-dependent is friction you feel daily.

Finding Mislocation: Right Bug, Wrong Line

BrassCoders anchors every finding to an exact file and line; the benchmark found the opposite for the models, reporting that all three mislocate issues at line-or-column granularity because of BPE tokenization. A model can name the right bug and still point at the wrong place.

The paper’s words: all language models “mislocate issues at line-or-column granularity due to tokenisation artefacts.” The cause it names is byte-pair encoding, the step that splits source text into tokens before the model ever reasons about it. Token boundaries don’t line up with line boundaries, so the model’s sense of where it is drifts. A finding you can’t locate is a finding a developer has to re-derive by hand, and it can’t drive a line-anchored CI rule or an automatic fix. Location isn’t a detail for a gate. It’s the whole interface.

Run-to-Run Non-Determinism, Measured

BrassCoders returns byte-identical findings on the same commit every run. LLM reviewers don’t, and two studies put numbers on it: a code-review study of four models over 70 commits, repeated five times at temperature zero, found the responses varied every time, and a code-generation study of ChatGPT found 47 to 76 percent of tasks produced no two identical outputs.

The code-review study, Measuring Determinism in Large Language Models for Software Code Review by Klishevich and colleagues (2025), tested GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and LLaMA 3.2 90B Vision. Its finding: “even with temperature minimized, LLM responses varied to different degrees,” which the authors call an inherently limited test-retest reliability.

The code-generation study, An Empirical Study of the Non-determinism of ChatGPT in Code Generation by Ouyang and colleagues (peer-reviewed, ACM DOI 10.1145/3697010), ran 829 problems and reported that the share of tasks with zero identical outputs across requests was 75.76, 51.00, and 47.56 percent on its three benchmarks. Its verdict on the common workaround: “setting the temperature to 0 does not guarantee determinism.” The reason sits below the API, where floating-point math and GPU batch effects at inference can select different tokens for the same input. A gate that returns a different answer on the same code isn’t a gate. It’s a suggestion.

Why These Numbers Disqualify the LLM as a Gate

BrassCoders exists to be the gate because a gate needs two properties the data shows LLMs lack: an exact location for every finding, and the same answer on every run. High recall can’t rescue a check that can’t be reproduced or pinned to a line.

Walk the failure modes. A reviewer that flips between a clean pass and three issues on identical input teaches the team one lesson: ignore it. A flag with no reliable line can’t auto-block a merge or feed an auto-fix. A precision near 0.75 means an auto-blocking gate would reject good code about a quarter of the time. Each of these is survivable for a second read; none is survivable for an automated gate. The deterministic scanner runs the other way, with lower recall but the same line-anchored findings on every run. Reproducibility and location are exactly the properties a gate is made of.

The Pattern the Data Points To

BrassCoders runs its deterministic scan on every commit and hands structured YAML to an AI assistant like Claude Code or Cursor for context-aware triage — the split the benchmark’s own authors recommend: models early for broad triage, deterministic scanners for high-assurance verification.

That recommendation is a direct quote from the paper: use language models “early in development for broad, context-aware triage, while reserving deterministic rule-based scanners for high-assurance verification.” BrassCoders is the reserving-deterministic-scanner half. It bundles 12 scanners — Bandit, Pylint, Pyre/Pysa, Semgrep, ast-grep, detect-secrets, plus six custom detectors — and every finding it emits carries a file, a line, a severity, and an evidence string that reads the same on every run.

The model’s recall doesn’t get thrown away. It gets applied after the reproducible gate, on a YAML file the assistant reads instead of on raw source it has to re-scan. BrassCoders reports the pattern; the assistant judges the context. That’s the division the numbers keep pointing at.

BrassCoders’ OSS core is Apache 2.0, free, and makes zero outbound network calls. The Paid plan adds AI-powered enrichment for $12 a month. Point it at a project and read the YAML it writes:

pip install brasscoders
brasscoders scan /path/to/your/project