AI Coding Assistant Vendor Risk: What Your Security Team Should Ask

Security teams evaluating AI coding assistants tend to collapse two distinct risks into one vendor conversation. The result: a thorough review of the vendor’s privacy policy and a gap on the security of what the AI actually produces. Getting both right requires different question sets — one for the assistant’s data practices, one for the code it generates.

One surface is the data-handling layer: what the vendor does with your source code during a session, whether that code feeds model training, and what deployment options exist for regulated environments. The other is the code-output layer: what the AI generates and what security issues that code contains before it reaches your repository.

Both surfaces require scrutiny. Conflating them leaves one unexamined.

Two Surfaces, Two Question Sets

BrassCoders addresses the code-output layer (the security issues in the code the AI assistant produces). The vendor risk checklist below addresses the data-handling layer: what the vendor does with your code during and after a session. These are different surfaces, and the questions don’t overlap.

Most security evaluations focus on the data-handling layer because it comes with documentation. The vendor has a privacy page, a data-processing agreement, and often a commercial buyer FAQ. That documentation is readable and auditable before you commit to a contract.

The code-output surface has no policy page. It has a finding rate: how often the AI generates code with security issues before those issues land in your repository. CISA’s Guidelines for Secure AI System Development, published in November 2023 and co-signed by cybersecurity agencies from the US, UK, Canada, Australia, and New Zealand, explicitly calls for static analysis in the build pipeline for AI-generated code. That call applies regardless of which AI assistant produced the suggestion.

A security team that reviews only the vendor’s privacy page has evaluated one surface. The sections below cover the vendor’s data-handling surface. The final section covers the code-output surface and what BrassCoders does there.

Questions About Code Transmission

BrassCoders’s OSS core sends zero bytes off the machine during a scan. The findings YAML writes locally, and no source code or scanner data leaves the developer’s machine in the free tier. For AI coding assistants, the transmission picture differs by design: the product’s core function requires sending code to a vendor server to generate a suggestion.

The questions to ask:

• Does the assistant send my full source file to vendor servers, or only the context window around the current edit?
• Is there an on-machine or offline mode where suggestion generation happens locally?
• What does the vendor retain from code sent during a session, and for how long?
• Does a data-processing agreement cover code transmission, or only account data and telemetry?

The answers vary significantly by vendor and often by pricing tier. Some assistants transmit the active file plus open files in the same project. Others transmit a narrower window. Retention periods range from session-only to several months. Before deploying to any environment with regulatory or contractual data-handling obligations, confirm the actual data flow against a DPA that covers it.

One distinction worth pressing on: a standard terms-of-service agreement and a data-processing agreement are not the same document. A ToS governs the vendor relationship; a DPA governs how data processed on your behalf is handled. If the vendor doesn’t offer a DPA, or the DPA doesn’t explicitly cover code sent during suggestion generation, the data-handling obligations for that code may fall outside any contractual protection — even if the privacy policy sounds reassuring.

Questions About Training Data Opt-Out

BrassCoders does not use customer scan data for model training. BrassCoders Paid sends already-redacted findings and a project signature of at most 7,500 characters of non-sensitive metadata to the enrichment gateway. Raw source code never leaves the machine, which means there is no source code available to include in any training corpus.

For AI coding assistants, the training question is distinct from the transmission question. A vendor may retain code only for session purposes and still use it to improve the model. The questions:

• Can my code be used for model training, and is opt-out the default or must I request it explicitly?
• Does the opt-out apply to fine-tuning as well as pretraining?
• Does the opt-out apply on all subscription tiers, or only on higher-tier plans?
• Does the opt-out apply retroactively to code already transmitted, or only to future sessions?

Default opt-in to training is common at entry-level price points. Commercial contracts often carry different data-handling terms than the standard user agreement. Teams working on proprietary or regulated codebases should confirm which terms govern their specific tier — and verify that the commercial buyer FAQ describes their actual agreement, not a representative scenario — before any code reaches the vendor’s servers.

Questions About On-Premises and Offline Options

BrassCoders’s --offline flag provides hard enforcement: brasscoders --offline scan exits non-zero if any network call is attempted at runtime. The zero-transmission guarantee is testable in CI, not just stated in a policy document.

For air-gapped environments and organizations with strict data residency requirements, the deployment model is often the first evaluation gate. The questions:

• Is an on-premises deployment available, and does it require a separate license or a different pricing tier?
• Is there an air-gapped option that makes zero outbound network calls during inference?
• For cloud deployments, which data residency regions are available, and is regional selection standard or an add-on?
• What model serving happens server-side, and what computation runs client-side during a suggestion?

Cloud-only vendors with no offline option may not meet FedRAMP requirements or industry frameworks that place conditions on where data processing occurs. The NIST AI RMF GOVERN function asks organizations to document AI system limitations — and a deployment model that rules out regulated environments is a limitation. Get the deployment answer before other evaluation criteria apply.

Questions About the AI-Generated Code Output

BrassCoders covers the code-output layer — the Python and TypeScript files the AI assistant produces that enter your repository — with 12 scanners, running entirely on the local machine.

Most vendor evaluations stop before reaching this question. The vendor’s privacy page describes what the vendor does with your code. It does not describe what security bugs that code contains. The questions to ask:

• Does the vendor run static analysis on generated code before surfacing it as a suggestion?
• What categories of security issues does the vendor’s internal tooling check for?
• Has the vendor published a finding rate for security bugs in AI-generated code?

For context on what an independent finding rate looks like: the BrassCoders published benchmark covered 15 real Python files from public AI-generated code repositories. BrassCoders caught issues in 9 of the 15 files, covering 11 of 12 bug categories the study examined. Zero proactive security warnings were issued during code generation by the AI assistants that produced those files. That benchmark represents one corpus. The question for any vendor: do they publish a comparable number, and against what test set?

Finding rates that are not published cannot be evaluated. If a vendor has no published number, the follow-up question is whether they run any independent third-party security testing on generated code — and whether those results are available to customers under contract. A vendor with no answer to either question has given you useful information about how they treat the code-output surface.

How BrassCoders Addresses the Code-Output Layer

BrassCoders runs 12 scanners against the code the AI produces — the part no AI vendor’s internal tooling reviews before the suggestion lands in your editor, since the suggestion displays before any commit-time gate runs.

The upstream scanners are Bandit, Pylint, Pyre/Pysa, Semgrep, ast-grep, and detect-secrets — the foundational tools for Python security analysis, code quality, taint flow, cross-language pattern matching, and credential exposure. Each is a mature open-source project with documented coverage.

The custom layer adds six more detectors: secret patterns, PII handling, phantom-API call detection for AI-generated imports that don’t exist, performance issues, content moderation, and JavaScript/TypeScript.

The JavaScript/TypeScript scanner uses Babel AST parsing and covers .js, .jsx, .ts, .tsx, .mjs, and .cjs files.

Caught patterns include eval() calls, innerHTML XSS sinks, document.write usage, hardcoded API keys, and hardcoded passwords.

The OSS core is Apache 2.0 licensed, requires no account, and sends nothing off-machine. Install with pip install brasscoders and run brasscoders scan .. Output writes to .brass/ai_instructions.yaml. Add brasscoders scan to your CI pipeline to gate every commit — a critical finding returns non-zero and blocks the merge.

For stricter data environments, brasscoders --offline scan enforces zero network calls at the scanner level. The offline guarantee is enforced in code, not in policy.

The vendor risk questions in the sections above address the AI coding assistant’s data practices: what they do with your code on their servers. BrassCoders gates on what the assistant actually produces: what the generated code contains once it’s back in your repository. These checks operate at different points in the workflow, against different surfaces, with different failure modes.

Run both. Neither substitutes for the other.