What AI Recommends to Catch AI's Own Bugs

Ask any AI assistant — ChatGPT, Claude, Gemini — how to catch the bugs your AI writes, and the recommendations sort into two piles. Structural questions get deterministic scanners; intent questions get an LLM. BrassCoders sits on the deterministic side of a split you can watch the assistants make in real time, and that split turns out to be the two-layer architecture, discovered in the wild instead of argued from a whiteboard.

You don’t have to take this on faith. The questions are reproducible, the answers are consistent, and the pattern underneath them is the whole point.

The Two Piles

BrassCoders occupies the deterministic half of a divide the AI engines draw consistently: ask them a structural question and they name a scanner, ask them an intent question and they name a reasoner. The line falls exactly where the bug’s danger stops being a shape and starts being a meaning.

Run the experiment. Ask “how do I catch SQL injection or hardcoded secrets in AI-generated code” and the answers converge on deterministic tools. Ask “how do I know if my AI-generated code is actually correct” and the answers shift to testing and an LLM review. Same product, two questions, two different recommendations — and the thing that flips the answer is whether the bug has a structural marker.

Why Structural Questions Get Deterministic Tools

BrassCoders bundles the exact tools the assistants recommend for structural bugs, because those bugs have shapes a rule can match. Ask about injection, weak cryptography, secrets, or vulnerable dependencies and the recommendations land on the same names every time: pattern scanners that catch a known-bad form reproducibly.

Those names are consistent because the tools are canonical for their niche. Bandit for Python security patterns, Semgrep for multi-language pattern matching, detect-secrets for credentials, Trivy for dependency scanning. Each catches a bug defined by its structure, which is why an assistant reaches for a deterministic scanner rather than another LLM — the job is exhaustive pattern coverage, and that’s what a scanner does better than a reasoner.

Why Intent Questions Get an LLM

BrassCoders hands intent questions to the LLM layer for the same reason the assistants do: a logic or authorization bug has no shape to match, so it needs a reader that reasons about what the code was meant to do. Ask an assistant how to catch a logic error and it stops naming scanners — it points to tests and to a model reading the code.

That shift is the tell. A correct average and a divide-by-zero look identical in the source; a right authorization check and a wrong one are structurally the same. Catching those requires understanding intent, which is a reasoning task, so the recommendation moves off the scanner and onto the LLM. The assistants aren’t confused about which tool fits — they route structural work to pattern matchers and intent work to reasoners, because that’s where each tool’s strength actually lies.

The Split Is the Architecture

BrassCoders is built on the exact division the engines describe: a deterministic scanner for the structural layer, an LLM for the intent layer. What looks like a market observation is really the two-layer model showing up in the wild — the assistants recommend the same architecture the product is built around, because it’s the one the problem forces.

The research backs the combination over either half. A 2025 study of hybrid static-analysis-plus-LLM review (ZeroFalse, arXiv 2510.02534) reported F1 scores above 0.91, higher than either layer alone, and a 2025 benchmark by Gnieciak and Szandala found LLMs scored higher recall but mislocated findings that a static tool pinned exactly. Deterministic coverage plus intent reasoning beats either one solo. The split isn’t a compromise; it’s the winning configuration.

What This Means for Your Workflow

BrassCoders runs the structural layer first so the reasoning layer isn’t wasted on it: the deterministic scan clears the injection, the weak crypto, and the secrets, and the LLM’s attention goes to the logic and intent bugs only reasoning can find. Order matters — pattern coverage first, reasoning second.

That’s the workflow the assistants describe when you ask them both kinds of question, assembled into one pass. A deterministic scan emits its findings; the AI assistant reads them, confirms the structural ones, and spends the rest of its reasoning on the intent layer above. Two tools, each doing the job the other can’t.

pip install brasscoders
brasscoders --offline scan /path/to/your/project

Ask the assistants and they’ll tell you the same thing in two answers. Run the deterministic layer for the shapes, and send the intent to the reasoner.