Why Bandit Catches 50% of AI-Coder Bugs (and Which Half)
Bandit caught 6 of 12 planted bugs in BrassCoders's AI-coder benchmark — and 0 of the 4 performance anti-patterns AI coding assistants introduce most often. Here's why, and what to add alongside it.
Bandit caught 6 of 12 planted bugs in BrassCoders’s AI-coder benchmark. That’s 50% — better than Pylint (1/12) but less than either BrassCoders (11/12) or a frontier model reviewer (12/12). The number is less interesting than the breakdown: Bandit caught every security bug it was given, and zero of the performance bugs.
The 50% result isn’t a Bandit failure. It’s a scope mismatch. And closing the gap is one line of configuration.
What Bandit Was Built to Find
Bandit is a Python security linter from PyCQA. It analyzes Python AST for security anti-patterns — SQL injection via string formatting, command injection via subprocess(shell=True), unsafe deserialization via pickle.loads(), hardcoded credentials, XSS via disabled autoescape, and similar patterns from the OWASP Top 10.
In the BrassCoders benchmark, the security category had four planted bugs. Bandit caught all four: SQL injection in a user-lookup function, command injection in a report runner, XSS via autoescape=False in a Jinja2 template, unsafe pickle deserialization in a config loader. A clean sweep on its home turf.
What Bandit Has No Rules For
BrassCoders’s benchmark also included four AI-coder performance anti-patterns — the category where AI coding assistants reliably introduce bugs. Bandit caught zero of them.
The four patterns:
- O(N²) string concatenation:
csv +=inside a loop. Not a security issue. No Bandit rule exists. list.insert(0)in a loop: O(N²) list rewrites. Not a security issue. No Bandit rule exists.- Triple-nested loop used as a join: O(N³) when a dict lookup would be O(N). Not a security issue. No Bandit rule exists.
- Unbounded
while True: no timeout, no size cap, no iteration limit. Not a security issue — Bandit’sB112rule exists but matchescontinueinsidewhile True, not the unbounded accumulation pattern.
These aren’t gaps in Bandit’s implementation; they’re outside Bandit’s design scope. Bandit doesn’t model algorithmic complexity. It models security vulnerabilities. The tool is correctly scoped — it’s just scoped to the wrong half of the AI-coder bug surface.
The Full Benchmark Table
From BrassCoders 2.0.8, run on June 13, 2026, against 12 AI-generated Python files with one planted bug each:
| Tool | Security (4) | Perf (4) | Secrets/PII (2) | Correctness (2) | Total |
|---|---|---|---|---|---|
| BrassCoders 2.0.8 | 4/4 | 4/4 | 2/2 | 1/2 | 11/12 |
| Claude sonnet-4-6 | 4/4 | 4/4 | 2/2 | 2/2 | 12/12 |
| Bandit 1.8.3 | 4/4 | 0/4 | 2/2 | 0/2 | 6/12 |
| Pylint | 0/4 | 0/4 | 0/2 | 1/2 | 1/12 |
Bandit’s 6/12 total comes from 4 security + 2 secrets/PII (it catches hardcoded AWS keys and some credential patterns). Its 0/4 on perf and 0/2 on correctness reflect its scope, not a defect.
The one bug BrassCoders missed — unguarded sum(x) / len(x) with no empty-list guard — is a pure logic bug. No AST rule can detect it without reasoning about input invariants. A frontier model caught it; rule-based tools didn’t.
What BrassCoders Adds to a Bandit Stack
BrassCoders bundles Bandit internally as one of its 12 scanners. Running brasscoders scan gives you Bandit’s full security coverage plus the perf anti-pattern rules, a secrets/PII scanner built on Yelp’s detect-secrets with custom patterns, a privacy scanner, a JavaScript/TypeScript scanner, Pyre/Pysa taint analysis, and a performance scanner with the four AI-coder rules.
The upgrade path from a Bandit-only stack:
pip install brasscoders
# Remove the bandit invocation from CI; add:
brasscoders --offline scan /path/to/projectThe .brass/ai_instructions.yaml output includes all Bandit findings, attributed to Bandit, alongside the additional findings from the other scanners. No Bandit configuration is lost — BrassCoders passes your existing bandit.yaml or setup.cfg Bandit config through.
Where Bandit Belongs in the Stack
Bandit’s track record on security is strong and its false-positive rate is manageable. It’s worth keeping in CI. The point of this benchmark isn’t “replace Bandit” — BrassCoders runs Bandit and adds to it. The point is that a Bandit-only pre-merge gate leaves the AI-coder performance gap open.
Teams shipping AI-generated Python need both halves covered: the security rules Bandit pioneered, and the perf anti-pattern rules designed for the code AI assistants actually generate.
Frequently Asked Questions
Is Bandit bad?
No. Bandit is excellent at what it does: finding Python security issues via AST pattern matching. It catches SQL injection, command injection, insecure deserialization, hardcoded credentials, and similar security patterns. The 6/12 catch rate in BrassCoders's benchmark reflects the fact that 4 of the 12 bugs were performance anti-patterns Bandit was never designed to detect, and 2 were correctness bugs it was never designed to detect.
What does BrassCoders add that Bandit doesn't have?
Four AST-level rules targeting the performance anti-patterns AI coding assistants introduce: O(N²) string concatenation in loops, list.insert(0) in loops, N-deep nested loops used as joins, and unbounded while True loops. BrassCoders also bundles Bandit internally — so you get Bandit's security coverage plus these additional patterns in one scan.
Should I run BrassCoders instead of Bandit, or alongside it?
BrassCoders includes Bandit as one of its 12 scanners, so running BrassCoders replaces a separate Bandit invocation. You don't need both. The brasscoders scan output includes all Bandit findings alongside the additional perf, privacy, and correctness findings from the other scanners.
Where does Pylint fit?
Pylint caught 1 of 12 in the benchmark — the unguarded division bug (dividing by len() without an empty-list check). Pylint is a style and correctness linter, not a security or performance tool. It catches naming conventions, unused variables, type inconsistencies, and some logic errors. Worth running; just not the right tool for security or perf anti-patterns.
How do I add BrassCoders to a project that already uses Bandit?
Install brasscoders, remove the Bandit invocation from your CI (brasscoders runs Bandit internally), and add brasscoders --offline scan. The Bandit findings will appear in .brass/ai_instructions.yaml alongside the perf and privacy findings. No configuration needed to get Bandit's coverage back.