Secure LLM Application Development
The security bugs that appear when your code calls AI — and how to catch them before production.
Building an LLM application creates a new security surface. The code that constructs prompts, routes user input to models, handles model output, and calls external tools carries vulnerabilities that don't appear in traditional web apps — and several that do. BrassCoders applies the same 12 scanners to LLM application code as to any Python codebase. The patterns shift; the detection layer doesn't.
This page is the canonical reference for the LLM application security question: what breaks, why, and what the scanner catches.
The Pattern Set Shifts in LLM Applications
BrassCoders finds four recurring security patterns in AI-generated LLM application code. Each is structural — the scanner catches it without reading the application logic or the model's output.
| Pattern | Why it appears | BrassCoders catch |
|---|---|---|
| Hardcoded AI provider API key | AI assistant fills credential field with plausible-looking key | detect-secrets (OpenAI, Anthropic key formats) |
| Missing input validation before prompt | Prompt construction treats user input as trusted | Bandit injection rules; Semgrep |
| Unescaped model output in templates | AI-generated Jinja2 code omits autoescape | Bandit template rules |
| SSRF via tool-call URL construction | Model-returned URL fetched without allowlist validation | Bandit HTTP request rules |
The LLM app scanning overview post walks through the full pattern set and what's outside the scanner's scope.
SSRF in AI Tool Calls
BrassCoders flags server-side request forgery risk when user-controlled or model-returned input flows into HTTP request construction without allowlist validation. In LLM tool-calling code, the attack surface is new: the model returns a URL as part of its response, the application fetches it, and an attacker who influences the model output routes the request to cloud metadata endpoints (169.254.169.254), internal services, or external targets.
OWASP A10:2021 (Server-Side Request Forgery) documents the class. The SSRF in AI tool calls post covers the specific tool-calling attack path, what BrassCoders catches structurally, and what requires the AI triage layer.
Mass Assignment in AI-Generated APIs
BrassCoders catches over-exposure patterns when FastAPI endpoints accept Pydantic models and write every field directly to the database. AI assistants generate .update(**request.dict()) because it satisfies the "update this user" prompt with the simplest possible implementation — without excluding admin-only fields like is_admin, role, or account_tier.
OWASP API Security Top 10 2023 API3 (Broken Object Property Level Authorization) covers this class. The mass assignment post explains the attack path and the explicit-field-inclusion fix.
XSS Through Model Output
BrassCoders flags unsafe template rendering in Python web applications — specifically Jinja2 templates with autoescape=False or missing autoescape configuration. In LLM applications, the risk is compounded: model output containing user-influenced content gets rendered to HTML without escaping, and a user who can influence the prompt can inject arbitrary HTML into the response.
The XSS through model output post covers the attack path and what Jinja2's autoescape=True setting does and doesn't prevent.
SQLAlchemy's Raw Query Escape Hatch
BrassCoders flags SQL injection via Bandit B608 when string formatting appears inside query strings — including SQLAlchemy's text() API used with f-strings. SQLAlchemy's ORM layer parameterizes queries by default; the risk is the escape hatch. AI assistants reach for session.execute(text(f"...{value}...")) when a complex query is faster to express in raw SQL than as an ORM expression.
The SQLAlchemy raw query post explains why the ORM usually protects you, what the escape hatch looks like, and the parameterized-bind-parameter fix.
OWASP LLM Top 10: What Static Analysis Covers
BrassCoders maps to a specific subset of the OWASP LLM Top 10 — the categories with source-level patterns a static scanner can match. LLM02 (Insecure Output Handling) maps to the template rendering and output sanitization rules. The categories that require runtime controls — LLM01 prompt injection, LLM03 training data poisoning, LLM08 excessive agency — are explicitly outside BrassCoders's scope and documented as such.
The OWASP LLM Top 10 for builders post gives the full mapping — what BrassCoders catches, what it doesn't, and a practical implementation order.
Prompt Injection: What BrassCoders Cannot Catch
BrassCoders does not catch prompt injection — and says so plainly. Prompt injection is a runtime attack on the AI agent, not a pattern in committed source code. Malicious instructions embedded in a README, a GitHub issue, or a dependency's documentation get followed by the agent without any source-level signature. The mitigations live in agent permissioning, sandboxing, isolated context windows, and egress control.
The prompt injection post covers the attack taxonomy and what the runtime mitigations actually look like. The agentic AI code security research category indexes the primary sources.
Data Leakage in LLM Application Code
BrassCoders's privacy scanner flags PII flowing across call paths — logging statements that capture request bodies with personal data, unencrypted storage of sensitive fields, and credential patterns in configuration files. LLM applications add a new leakage surface: user input that contains personal data flows into the prompt, which flows into the model, which returns it in a context the application might log.
The LLM data leakage post covers the structural patterns BrassCoders flags and what the AI triage layer handles.
All Secure LLM Application Posts
- Scanning LLM Application Code — overview of what BrassCoders finds in AI-app codebases
- SSRF in AI Tool Calls — model-returned URLs, cloud metadata, allowlist validation
- Mass Assignment in AI-Generated APIs — FastAPI, Pydantic, is_admin fields
- Cross-Site Scripting Through LLM Output — Jinja2 autoescape, model output rendering
- SQLAlchemy's Raw Query Escape Hatch — text() with f-strings, Bandit B608
- OWASP LLM Top 10 for Builders — static analysis coverage map
- Prompt Injection in AI Coding Agents — what BrassCoders can't catch; runtime mitigations
- Where LLM Apps Leak Data — PII leakage patterns; privacy scanner coverage
Frequently Asked Questions
Can BrassCoders scan an LLM application codebase?
Yes. BrassCoders applies the same 12 scanners to LLM application code as to any Python codebase. The patterns shift — hardcoded AI provider API keys, missing input validation before prompt templates, missing output sanitization, tool-call SSRF — but the detection layer is the same. The key constraint: prompt injection is a runtime attack on the agent, not a source pattern, and static analysis cannot detect it.
What is SSRF in AI tool-calling code?
Server-Side Request Forgery in LLM tool calls occurs when the model returns a URL or path as part of its response and the application fetches it without validating whether it points to an internal resource. An attacker who can influence model output can route requests to cloud metadata endpoints (169.254.169.254), internal services, or arbitrary external hosts. BrassCoders flags request-construction patterns where user-controlled or model-returned input flows into HTTP calls without allowlist validation.
What is mass assignment in a FastAPI application?
Mass assignment is when a FastAPI endpoint accepts a Pydantic model and writes every field directly to the database record — including is_admin, role, or account_tier — because the model binding is simpler to write than explicit field selection. OWASP API Security 2023 API3 (Broken Object Property Level Authorization) covers this category. An attacker sends a crafted request that sets is_admin=true and the application accepts it.
Does BrassCoders detect prompt injection?
No, and it will not claim to. Prompt injection is a runtime attack on the AI agent — malicious instructions embedded in external content the model reads (a README, an issue, a web page). There is nothing in committed source code for a static scanner to match. BrassCoders scans source deterministically; prompt-injection mitigations live in agent permissioning, sandboxing, and isolated context windows.
How does XSS happen through LLM output?
When an LLM application renders model output to HTML without escaping, a user who can influence the prompt can inject arbitrary HTML. The attack path: user input → model → unescaped template rendering → XSS. Jinja2's autoescape=True prevents it; many AI-generated Flask and Jinja2 templates set it to False or omit it. BrassCoders flags autoescape=False and related unsafe rendering patterns.
What is the OWASP LLM Top 10?
OWASP's Top 10 for Large Language Model Applications is the standard taxonomy of risks in LLM-integrated systems, analogous to the OWASP Web Application Top 10. It covers prompt injection (LLM01), insecure output handling (LLM02), training data poisoning (LLM03), model denial of service (LLM04), supply chain vulnerabilities (LLM05), sensitive information disclosure (LLM06), insecure plugin design (LLM07), excessive agency (LLM08), overreliance (LLM09), and model theft (LLM10). Static scanning addresses a subset of these; the rest require runtime controls.
What does BrassCoders send off my machine when scanning an LLM application?
Nothing by default. brasscoders scan runs offline — zero outbound network calls, no source code transmitted. The BrassCoders Paid plan sends scanner findings (already redacted at source) and a short project signature to the hosted gateway; it never transmits raw source code. The full data inventory is at coppersun.dev/what-api-review-sends-off-your-machine/.