Scan, Patch, Re-Scan: Verifying AI Bug Fixes

Your AI assistant wrote the bug. Now it’s proposing the fix, and if you let it, confirming its own fix worked. That last step is where the loop breaks. A verifier has to be independent of the thing it checks, and a language model grading its own patch is neither independent nor repeatable. BrassCoders, the scanner that catches what AI assistants structurally miss, is the independent check: it scans deterministically, so the same code yields the same findings every run, and a re-scan after the patch tells you whether the finding is actually gone.

The loop has three moves: scan, patch, re-scan. The third move is the one most AI workflows skip.

Why an LLM Can’t Verify Its Own Fix

BrassCoders re-scans the patched file with the same rules that produced the original finding, so the verification comes from a system that took no part in writing the fix. An LLM asked to confirm its own patch samples a new answer each time, and the same diff can read as fixed on one run and unresolved on the next.

None of this is a criticism of the model’s coding. It reflects what the two tools are. A language model generates text by sampling from a distribution, so its output shifts between runs. A rule-based scanner matches patterns in an abstract syntax tree and returns the same hits for the same input. When the writer and the checker are the same sampling process, a green light means the model didn’t happen to flag the problem this time, not that the problem left the code.

The measurement backs this up. A 2025 benchmark, Large Language Models Versus Static Code Analysis Tools by Damian Gnieciak and Tomasz Szandala, ran three LLMs against three static analyzers on real vulnerabilities. The models scored higher on F1 through recall, yet they mislocated findings at line-and-column granularity because of tokenization. The authors’ recommendation lands on the loop directly: use language models early for broad, context-aware triage, and reserve deterministic rule-based scanners for high-assurance verification.

Step 1: Scan and Read the Findings File

BrassCoders writes its findings to .brass/ai_instructions.yaml, a severity-ranked YAML file that carries a file path, line number, and detecting scanner for every finding. The command brasscoders --offline scan . produces that file with zero outbound network calls.

pip install brasscoders
brasscoders --offline scan /path/to/your/project

The scan runs BrassCoders’ 12 bundled scanners against the project and collapses their output into one ranked file. Open .brass/ai_instructions.yaml and the executive_summary block gives you a risk_level and a prioritized recommendation. The security_critical block lists each finding with its file:line and a remediation pointer. That’s the input your assistant needs.

Step 2: Hand the Findings to Your AI Assistant

BrassCoders formats .brass/ai_instructions.yaml for an AI assistant to read directly, no copy-paste of source required. An assistant like Claude Code or Cursor opens the file, reads the ranked findings with their file:line and remediation pointers, and proposes patches.

The division of labor is the point. BrassCoders is the deterministic pattern reporter. The assistant is the context-aware layer that reads the finding against the surrounding code and decides how to fix it. For a SQL injection built by string formatting, the assistant swaps the format for a bound parameter. For a hardcoded token, it moves the value to an environment variable. BrassCoders doesn’t guess intent, and it doesn’t write the patch. It reports what the rules matched and lets the assistant reason.

Step 3: Re-Scan to Verify the Fix

BrassCoders verifies the fix the same way it found the bug: run brasscoders --offline scan . again, and the SQL injection finding that flagged app.py either clears from .brass/ai_instructions.yaml on the next run or it doesn’t. Same input, same findings, so a cleared finding means the pattern left the code, not that a model returned a kinder answer.

Run the loop until the re-scan comes back clean on the findings you meant to fix. A residual finding after a correct patch happens, and it’s informative. Semgrep’s taint rule, for one, keeps firing on a correctly parameterized SQLite call because it tracks user-controlled data to the execution sink without distinguishing the bound-parameter form. That’s where the assistant reads the surviving finding, sees the parameter tuple in the code_snippet field, and marks it a false positive. The re-scan narrows the surface. The assistant closes it.

BrassCoders publishes a reproducible version of this drop. In its N=15 AI-code-findings corpus, a Flask endpoint generated from a one-line prompt ships SQL injection at line 22, and the first scan reports three findings on that line from two independent detectors plus Bandit. Apply the parameterized ? placeholder and re-scan, and the input-validation and Bandit findings clear. One Semgrep taint finding remains as the known false positive. Three findings become one triage decision, and the decision takes seconds. The corpus is Apache 2.0 and pinned to a commit, so the before and the after reproduce on your own machine.

Fail the Build When a Critical Survives

BrassCoders exits with a non-zero status (exit code 2) when you pass --fail-on-critical and any critical- or high-severity finding is present, so a CI step or a pre-commit hook fails the build deterministically. The exit code comes from the deterministic scan, so a build that fails today fails tomorrow on the same commit.

Add the flag to the scan step and the gate is one command:

brasscoders --offline scan --fail-on-critical .
# exit code 2 when a critical- or high-severity finding is present; 0 otherwise

For a custom threshold — a specific severity, or a count above some number — read the deterministic .brass/statistics.yaml instead of relying on the exit code:

brasscoders --offline scan .
python3 - <<'PY'
import yaml, sys
stats = yaml.safe_load(open(".brass/statistics.yaml"))
crit = stats.get("distribution", {}).get("by_severity", {}).get("critical", 0)
sys.exit(1 if crit else 0)
PY

A green run means no critical pattern survived the last re-scan. A red run points at the same file and line every time, so the team learns to act on it. That’s the difference a deterministic gate makes: the check means the same thing on Monday that it means on Thursday.

Point branch protection at the job and the gate becomes a real merge block. A pull request that reintroduces a critical pattern can’t merge until a re-scan comes back clean. The scanner ran the same rules on the same code, so nobody argues with the result.

The Hybrid Pattern the Research Points To

BrassCoders runs the deterministic detection pass and hands structured output to an AI assistant for judgment, the same split a 2025 framework called ZeroFalse reports reaching an F1 of 0.912 on the OWASP Java Benchmark and 0.955 on the OpenVuln dataset. Detection first, model judgment second.

ZeroFalse (Iranmanesh and colleagues, 2025) treats a static analyzer’s output as a structured contract, enriches each finding with flow-sensitive traces and CWE-specific knowledge, then has an LLM adjudicate whether it’s real. Recall and precision both land above 90% on the two benchmarks. The ordering is what matters: a deterministic tool decides what to look at, and the model decides what it means. The scan-patch-re-scan loop applies the same ordering to remediation. BrassCoders detects and re-verifies. The assistant reasons and fixes.

Run the Loop on Every Fix

BrassCoders makes the loop cheap enough to run on every change. The OSS core is Apache 2.0, free, and offline by default, so scanning and re-scanning cost nothing and send nothing off the machine.

pip install brasscoders
brasscoders --offline scan /path/to/your/project

Fix a finding, run the scan again, and read .brass/ai_instructions.yaml to confirm it’s gone. The check that clears your AI’s patch should be the one thing in the loop your AI didn’t write.