What Your Auditor Wants From Your AI Codebase

A SOC 2 auditor reviewing your AI-augmented engineering team asks one question first: how do you know what’s in the code? If the answer involves a stochastic scanner whose output varies between runs, the audit has a problem. If the answer involves a deterministic scanner whose output is reproducible against a pinned commit, the audit has a clean control.

BrassCoders, the bug scanner for AI coders, is the deterministic scanner. This piece walks through the audit posture an AI-augmented engineering team needs by 2026 — the SOC 2 controls BrassCoders maps to, what the YAML looks like to an auditor, and what BrassCoders is honestly not audited for.

This is the sixth piece in BrassCoders’s AI Coding Assistant Blind Spots pillar — the audit answer to the technical coverage map laid out in the rest of the series.

Why Auditors Reject Stochastic Output

BrassCoders detection output is bit-for-bit reproducible against a pinned commit, by design. The OSS core runs the same 12 scanners with the same configuration on the same input and produces the same finding list every time. Auditors accept this; auditors reject stochastic scanners that produce different findings on different runs.

The auditor’s logic. A control trail in SOC 2 needs evidence. Evidence has to be reproducible — the auditor needs to be able to re-run the check and see the same result. A scanner whose output varies between runs because the model temperature changed, because the training data refreshed, because the prompt template was tuned, cannot produce an audit trail. The auditor cannot tell which version of the scanner ran when the artifact was generated; the auditor cannot reproduce the artifact.

LLM-based PR reviewers are stochastic. Same input, different output. Useful for engineering, problematic for audit. Most teams that adopt LLM-based PR review keep the existing deterministic SAST around for the audit trail — even if the SAST finds fewer real bugs than the LLM, the SAST findings are evidence-grade and the LLM findings are not.

BrassCoders ships both layers and keeps them separate. The deterministic scanner layer (the OSS core, plus the same scanners in Paid) is the audit-grade evidence. The AI-powered enrichment layer (Paid only) is the developer productivity layer. The two layers serve different stakeholders. The auditor reads the scanner layer.

The SOC 2 Controls BrassCoders Maps To

BrassCoders maps cleanly to two SOC 2 Trust Services Criteria: CC8.1 (change management — every change is scanned) and CC7.1 (system operations — vulnerabilities are monitored). The mapping is direct enough that auditors recognize the artifact without extensive explanation.

CC8.1 — Change Management. The control requires the entity to authorize, design, develop, test, approve, and implement changes to infrastructure, data, software, and procedures. The BrassCoders scan output, persisted as a CI artifact tied to the commit hash, evidences the “test” and “approve” steps. Every pull request gets scanned; every scan produces a YAML; every YAML lives in the build artifact store. The auditor walks the trail.

CC7.1 — System Operations. The control requires the entity to monitor system components and operations for anomalies that are indicative of malicious acts, natural disasters, and errors. The BrassCoders scan history, retained per release, evidences ongoing vulnerability monitoring. The auditor reviewing CC7.1 sees a continuous trail of vulnerability detection runs against the production codebase.

The mapping for adjacent frameworks. HIPAA 164.308(a)(1) (security management process): same evidence applies. ISO 27001 A.12.6.1 (technical vulnerability management): same evidence. PCI-DSS requirement 6.6 (review of public-facing applications for vulnerabilities): applicable for the AppSec scope. The same scan trail covers multiple frameworks because the underlying control is the same — continuous deterministic vulnerability monitoring.

AICPA’s Trust Services Criteria defines the SOC 2 controls. The mapping above is BrassCoders’s reading; your auditor will translate to the specific control language your firm uses.

Reproducibility As An Audit Artifact

BrassCoders treats reproducibility as a load-bearing feature, not an implementation detail. The reproducible-benchmarks page ships scan results against nine open-source codebases with pinned commit hashes; any auditor or engineer can re-run the scan and confirm the same output.

The benchmarks page is at coppersun.dev/benchmarks.

The mechanic. BrassCoders pins every scanner version in pyproject.toml. Pylint, Bandit, Semgrep, detect-secrets — each one is locked to a specific minor version. Pyre is locked to a narrow window because the Pysa model format has been unstable across minors. The version pinning is intentional; bumping a scanner version is a calibration event that requires re-verifying the bundled detection model.

The reproducibility checklist for an audit. (1) Customer pins BrassCoders version in their lockfile. (2) Customer pins the project commit hash. (3) Customer runs brasscoders scan and persists the output as a CI artifact. (4) Auditor runs the same BrassCoders version against the same commit and confirms the output matches. The check completes in minutes against typical codebases.

The honest scope. Reproducibility is bit-exact at the scanner layer. The Paid plan’s AI-powered enrichment layer is deterministic at the scanner layer but stochastic at the embedding layer — the same scanner findings can produce slightly different rankings across runs because embedding models do not promise bit-exact output. For audit purposes, the underlying scanner findings are what matters, and those are reproducible.

What The YAML Looks Like To An Auditor

BrassCoders writes the scan output as YAML in .brass/ai_instructions.yaml, designed to be human-readable by developers and machine-readable by AI assistants. Auditors read the same file the AI does; the structure is stable, the fields are documented, the schema lives in the BrassCoders repo.

The schema in summary. The top-level YAML contains a metadata section (scan timestamp, BrassCoders version, scanner versions, project root path), a critical_issues array (CRITICAL-severity findings, with file path, line number, finding type, and description per entry), a findings array (all other findings, sorted by relevance), and a summary section (counts by severity, counts by scanner, total finding count). The schema is documented at /legal/privacy for what the scanner persists and at the BrassCoders GitHub repo for the full structure.

The audit-relevant section. Auditors reviewing CC8.1 walk the critical_issues array per release. The expectation is that critical findings are addressed before release, not that critical findings are zero. The audit trail is: critical_issues at scan time → ticket opened → ticket closed → next scan shows critical_issues addressed. The YAML supports the trail.

What the YAML deliberately does not contain. No raw source code. No PII matched values (the privacy scanner redacts before serialization). No secret values (the secrets scanner records type and hash, never the secret itself). The two-boundary redaction policy is documented in the privacy disclosure. Auditors who confirm the redaction policy confirm that the scan output is safe to retain as a build artifact indefinitely.

What BrassCoders Isn’t Audited For (Honest Scope)

BrassCoders is not a third-party-audited security service, and any claim suggesting otherwise would be inaccurate. The OSS core is a local CLI with no service surface; there is no entity to audit. The Paid plan’s gateway is a thin proxy in front of a hosted embedding model; the gateway logic is documented but not third-party attested.

What this means in practice. A customer adopting BrassCoders for SOC 2 evidence uses BrassCoders as a deterministic vulnerability scanner. The deterministic detection layer is the evidence. The customer’s own auditors confirm the deterministic behavior by reproducing scans against pinned commits. The customer does not point at a BrassCoders-issued SOC 2 report because there is none.

What BrassCoders provides instead. Full documentation of the data plane in /legal/privacy. Reproducible benchmarks at /benchmarks. Open source code at the detection layer (Apache 2.0). Each of these is independently verifiable by the customer or their auditor. The verifiability replaces the third-party attestation; the auditor can confirm directly rather than rely on a report.

When BrassCoders will pursue SOC 2. The honest answer is when customers ask in writing. The cost of a SOC 2 Type II audit is roughly $50,000-$100,000 for an early-stage SaaS; BrassCoders will pursue the audit when paying customers cite it as a procurement blocker. As of mid-2026, the deterministic-by-design and source-available posture has been sufficient evidence for the customers in our pipeline.

Closing

BrassCoders ships deterministic, reproducible vulnerability detection across 12 scanners, packaged as YAML output that an auditor reads the same way an AI assistant does. The audit posture is the byproduct of the deterministic-by-design architecture; it is not a separate feature.

Install with one command:

pipx install brasscoders
brasscoders scan /path/to/project

Persist .brass/ai_instructions.yaml as a CI artifact. Tie the artifact to the commit hash. When the auditor asks how you know what’s in your code, point at the artifact.

The full coverage map of what BrassCoders detects — and what it does not — is in the AI Coding Assistant Blind Spots pillar. The seven blind-spot categories, the seven detector mappings, and the honest scope of each detector are documented there.