Scanning LLM Application Code: What BrassCoders Finds

LLM application code has a predictable security profile. The scanners don’t change — BrassCoders applies the same 12 to an AI app as to any Python project. What changes is the finding distribution: hardcoded provider keys dominate in early development, input validation disappears between user input and the prompt template, and SQL and shell injection reappear wherever tool calls exist. None of these require LLM-specific detection rules. All fall under existing scanner coverage, applied to code that follows a recognizable structure.

The LLM App Security Surface

BrassCoders treats an LLM application as a Python codebase with a known architecture: user input arrives, gets woven into a prompt, the prompt goes to an AI provider, and the response routes somewhere — a database, a shell, an API, a rendered page. That architecture concentrates risk at three joints: the credential used to reach the provider, the construction of the prompt from untrusted input, and the routing of the model’s output back into downstream systems.

Each joint maps to existing scanner coverage. Bandit catches shell injection and SQL injection. The detect-secrets scanner catches hardcoded credentials. The API security scanner flags missing input validation before string operations. No new rules are required. The architecture is predictable; the findings are predictable.

API Keys: The AI Provider Credential Pattern

BrassCoders detects 20+ secret formats — OpenAI keys, Anthropic keys, AWS access keys, GitHub personal access tokens, Stripe live keys, JWTs, PEM-formatted private keys, and high-entropy strings — using Yelp’s detect-secrets as the upstream library plus custom patterns added in the scanner layer. In LLM application code specifically, the AI provider key is the credential most consistently written inline during early development.

The pattern is easy to understand: a developer prototypes an OpenAI or Anthropic integration, puts the key directly in the source to get something working, and the key commits before the .env refactor happens. The scanner sees the sk- prefix and flags it as a HIGH severity finding. The AI consumer of the YAML output — Claude Code, Cursor, whatever your team uses — then confirms whether the pattern is a real credential or a placeholder, because that judgment requires context the scanner deliberately doesn’t attempt.

Yelp’s detect-secrets is the open-source entropy and pattern library that backs the secret detection layer. New credential formats arrive via library upgrade. For a deeper look at how this plays out in a reproducible benchmark, the hardcoded credentials post covers two real cases from the published corpus.

Input Validation Before the Prompt Template

BrassCoders flags missing input validation when user-controlled data routes directly into string operations without a sanitization step. In standard web applications, this catches SQL injection and XSS setup. In LLM application code, the same structural pattern appears at prompt construction.

A prompt template built by string concatenation or f-string interpolation from user input has the same shape as a SQL query built by concatenation: unvalidated input lands in a sensitive context. BrassCoders finds the structural absence — no type check, no length bound, no allow-list before the concatenation. What happens downstream when a crafted input reaches the model is a runtime question, not a static one. The scanner reports the missing validation. Your AI assistant’s triage layer decides whether the route is exploitable and what the fix should look like.

This is the BrassCoders division of labor in practice. The scanner doesn’t reason about what a malicious prompt could make the model do. It finds the open door; the AI assistant evaluates whether the door matters.

Output Sanitization After the Model Returns

BrassCoders catches a class of findings specific to how LLM apps handle the model’s response: the output is untrusted data, and the code should treat it that way. When a model response renders in a web page, gets written to a database, or routes to another system without sanitization, the same injection classes that apply to user input apply to what the model returned.

These are Bandit B701 and related findings, surfacing because the code treats the model’s output as clean rather than as an untrusted string. The scanner can’t know the model will behave well. It flags the assumption.

Tool Calls and the SQL/Shell Injection Risk

BrassCoders catches SQL injection and shell injection wherever user-controlled input reaches those execution surfaces — and in LLM application code, the route often passes through a tool-call layer. An agent that can query a database or run a shell command, built with user input in the call chain without sanitization, has the same vulnerability as any other application with the same structural gap.

Bandit’s B608 finding (possible SQL injection) and B602/B603 findings (shell invocation with untrusted input) fire on pattern. The scanner doesn’t trace the tool-calling framework’s routing. It finds the code that executes the query or the shell command and checks whether the input going in was sanitized. If it wasn’t, that’s the finding.

Two dedicated posts go deeper if this is your primary concern: SQL injection in AI-generated Python covers the Bandit B608 pattern in detail, and command injection with shell=True covers the shell invocation side.

What’s Outside the Scanner’s Scope

BrassCoders draws a clear boundary at the source layer. Two risks specific to LLM applications sit outside what any static scanner can reach, and knowing that boundary is part of what makes the scanner’s coverage trustworthy.

Prompt injection is a runtime attack on the AI agent itself, not a pattern in the source code the agent produces. A malicious instruction embedded in a retrieved document or user message can redirect the agent’s behavior at inference time. There’s nothing in the committed code for a scanner to match. Mitigations live in agent permissioning, sandboxed execution, isolated context windows, and egress control. The prompt injection post covers this in full.

Data leakage at inference — when the model emits sensitive information it was handed through context or retrieval — is similarly invisible to static analysis. The sensitive string is data flowing through the system at runtime, not a literal committed to the repository. BrassCoders catches PII and credentials written into source code; governing what the model returns is a runtime control at a different layer. The LLM app data leakage post covers where that boundary sits.

How to Run the Scan

BrassCoders installs via pip and requires no account for the OSS core.

pip install brasscoders
brasscoders scan /path/to/your/llm-app

The scan runs entirely locally. In OSS-core mode, zero bytes leave the machine — no telemetry, no outbound network calls. The findings land in .brass/ai_instructions.yaml, formatted for direct consumption by Claude Code or Cursor. Your AI assistant reads the YAML, source-verifies each finding, and triage begins.

BrassCoders requires Python 3.10+ and runs on macOS, Linux, and Windows (WSL2). Three machine activations come with each BrassCoders Paid license — the OSS core has no activation requirement and no usage caps.

For the full list of what data leaves the machine in Paid mode, the data-handling post covers the exact payload.