Does high test coverage mean the code is well tested?

No. Coverage counts which lines a test executed, not whether the test would notice a wrong answer. Inozemtseva and Holmes generated 31,000 test suites across 724,000 lines of code and found only a low-to-moderate correlation between coverage and fault detection once suite size is controlled for; they concluded coverage should not be used as a quality target. A test that calls a function and asserts nothing raises coverage exactly as much as one that checks every output.

Do AI-generated tests catch real bugs?

Often they don't. A 2023 study of ChatGPT-generated unit tests found only 24.8% executed without errors, and 17.3% compiled but failed on the model's own incorrect assertions. A separate study saw an LLM reach over 80% line coverage on the HumanEval toy benchmark but under 2% on the real-world SF110 codebase. AI test generators are good at raising the coverage number and much weaker at writing assertions that fail when the code is wrong.

What is coverage theater?

Coverage theater is a test suite that raises the coverage percentage while verifying little or nothing. Researchers cataloguing LLM-generated tests found named test smells including Empty Tests and Duplicated Asserts, both of which execute code and lift coverage without meaningfully checking behavior. The coverage report looks healthy; the suite would pass even if the code returned the wrong result.

How do I measure whether my tests actually verify behavior?

Mutation testing. A mutation tool seeds small faults into your code and reports how many your tests catch, which is a direct measure of fault-detecting power rather than execution. Just and colleagues validated the approach on 357 real faults across 321,000 lines and found mutant detection correlates with real-fault detection independently of coverage. PIT is the reference tool for the JVM; mutmut and Cosmic Ray are the common Python options.

Does BrassCoders check my tests?

No. BrassCoders does not run or grade your test suite; it scans the code the tests are supposed to protect. A static analysis pass flags a real SQL injection or unsafe deserialization call whether or not the tests cover that line, so a green, high-coverage suite full of hollow assertions can't hide a finding BrassCoders would report. Coverage and mutation score tell you about your tests; BrassCoders tells you about the code directly.

Will My AI Write Tests That Catch Bugs?

Coverage Counts Execution, Not Verification

BrassCoders treats a coverage percentage as an execution count, not a proof of correctness: coverage records which lines a test ran, never whether the test would fail if those lines returned the wrong answer. The resources below are the canonical evidence — the largest peer-reviewed study of coverage versus effectiveness, the measurements of what AI test generators actually produce, and the mutation testing that measures fault-detecting power directly.

📄 Coverage Is Not Strongly Correlated with Test Suite Effectiveness

Inozemtseva & Holmes — University of Waterloo, ICSE 2014 · cs.ubc.ca (ICSE 2014)

BrassCoders treats this ICSE 2014 paper as the peer-reviewed spine of the coverage-theater problem. The authors generated 31,000 test suites across five systems totaling 724,000 lines of code and found only a low-to-moderate correlation between coverage and a suite's fault-detection power once the number of test cases is controlled for. Their conclusion: coverage is useful for finding under-tested code, but it should not be used as a quality target, because it is not a good indicator of effectiveness.

What it's good for: the largest study to date of whether a coverage number predicts a suite's ability to catch bugs. Where BrassCoders draws from it: the evidence that a high coverage figure on AI-generated tests is not a signal the code is verified.

📄 Assertions Are Strongly Correlated with Test Suite Effectiveness

Zhang & Mesbah — University of British Columbia, ESEC/FSE 2015 · ece.ubc.ca (FSE 2015)

BrassCoders treats this FSE 2015 result as the constructive answer to what coverage misses. Composing 6,700 test suites from 24,000 assertions across five real-world Java projects, the authors found the number of assertions in a suite strongly correlates with its effectiveness, and that assertion coverage tracks effectiveness where line coverage does not. The lesson for AI-generated tests is direct: count the assertions, not the executed lines.

What it's good for: what actually predicts fault detection when line coverage does not. Where BrassCoders draws from it: the reason BrassCoders frames assertion-free AI tests, not low coverage, as the real failure.

📄 No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

Yuan, Lou, Liu, Ding, Wang, Chen, Peng — Fudan University, arXiv:2305.04207 (2023) · arxiv.org/abs/2305.04207

BrassCoders treats this 2023 study as the clearest measurement of assertion quality in AI-generated tests. Of 1,000 unit tests ChatGPT produced, 42.1% compiled and only 24.8% executed without errors; 17.3% were compilable but failed on the model's own incorrect assertions. A test that turns green on a wrong assertion is worse than no test, because it reports confidence the code has not earned.

What it's good for: the compile, execution, and assertion-correctness rates of AI-generated unit tests at scale. Where BrassCoders draws from it: the evidence that AI tests fail most often on the assertion, the part that decides whether a bug is caught.

📄 Using Large Language Models to Generate JUnit Tests: An Empirical Study

Siddiq, Santos, Tanvir, Ulfat, Al Rifat, Carvalho Lopes — EASE 2024 (arXiv:2305.00418) · arxiv.org/abs/2305.00418

BrassCoders treats this study as the evidence that AI test coverage collapses off the benchmark. A Codex model reached above 80% coverage on the HumanEval dataset but no model exceeded 2% coverage on the real-world SF110 benchmark, and the generated tests carried named test smells including Duplicated Asserts and Empty Tests. Builders should read the HumanEval-to-SF110 gap as the distance between a demo and a production codebase.

What it's good for: how AI test coverage differs between toy benchmarks and real-world code, and what smells the tests contain. Where BrassCoders draws from it: the reason a headline coverage figure from AI-generated tests deserves a second look, not trust.

📄 An Empirical Evaluation of Using LLMs for Automated Unit Test Generation

Schäfer, Nadi, Eghbali, Tip — IEEE TSE 2024 (arXiv:2302.06527) · arxiv.org/abs/2302.06527

BrassCoders treats this TSE 2024 evaluation as proof that AI generators are genuinely good at the coverage number, which is why it flatters them. Its TestPilot system, run on 25 npm packages with 1,684 API functions, reached a median 70.2% statement coverage and 52.8% branch coverage. Hitting coverage is the thing these tools do well; whether the tests verify behavior is the separate question the other research answers.

What it's good for: the coverage AI test generators can reach on real packages. Where BrassCoders draws from it: the context that coverage is the metric AI tests chase, which is exactly why coverage alone cannot grade them.

📄 Are Mutants a Valid Substitute for Real Faults in Software Testing?

Just, Jalali, Inozemtseva, Ernst, Holmes, Fraser — FSE 2014 · cs.washington.edu (FSE 2014)

BrassCoders treats this FSE 2014 paper as the validation that makes mutation score trustworthy. Studying 357 real faults across 321,000 lines of code in five applications, the authors found a statistically significant correlation between mutant detection and real-fault detection, independent of code coverage. That independence is the point: a suite's mutation score predicts real-bug catching in a way its coverage number does not.

What it's good for: whether seeded mutants are a valid stand-in for real faults when judging a test suite. Where BrassCoders draws from it: the evidence behind pointing teams to mutation testing, not coverage, to measure whether AI tests verify anything.

🔧 PIT — Real World Mutation Testing

PIT (pitest.org) — JVM mutation testing · pitest.org

BrassCoders points to mutation testing as the check coverage cannot perform, and PIT is its reference implementation on the JVM. PIT's own documentation states that traditional coverage measures only which code is executed and does not check whether tests can detect faults, then seeds faults and reports how many the suite kills. A suite that asserts nothing kills nothing, whatever its coverage. Python projects reach for mutmut or Cosmic Ray for the same measurement.

What it's good for: the primary-source definition of what mutation testing measures that coverage does not. Where BrassCoders draws from it: the measurement BrassCoders recommends alongside a scan to tell whether a test suite actually verifies behavior.

🔧 coverage.py — What Coverage Measurement Records

coverage.py (Ned Batchelder) — Python documentation · coverage.readthedocs.io

BrassCoders treats coverage.py as the primary source for what a Python coverage number actually means. Its documentation says the tool monitors which parts of the code have been executed and identifies code that could have been executed but was not. It makes no claim about whether executed code was checked for correct behavior. Coverage tells you what your tests touched, and stops there.

What it's good for: the exact, vendor-stated meaning of a Python line-coverage figure. Where BrassCoders draws from it: the grounding for treating coverage as an execution count, not a verification signal, in AI-written test suites.

Will My AI Write Tests That Catch Real Bugs?