AI-Generated Test Debt: Tests That Pass But Don't Test Anything

Green CI is not the same as tested code. It means your tests ran and didn’t crash. Whether they actually verified anything is a separate question — one that AI coding assistants have quietly made harder to answer.

AI assistants write tests fast. Too fast to think about what an assertion is actually checking. The result is a category of bug that doesn’t fail your suite: tests that exist, pass, and prove nothing. This post names the three most common patterns, shows the code, and is honest about what BrassCoders catches versus what requires different tooling.

The Always-True Assertion

BrassCoders surfaces always-pass assertion patterns via Pylint, flagging tautological comparisons and assert True statements found in test code during a standard scan. These are the obvious structural failures. The harder case — a syntactically valid assertion that always passes because it checks the wrong thing — is beyond what any static analyzer can catch reliably.

Here’s the pattern AI assistants produce most often:

def test_get_user_profile():
    response = client.get("/api/users/42")
    assert response is not None

That assertion passes whenever the function returns anything at all. It passes if the function returns a 404. It passes if it returns {"error": "user not found"}. It’s checking that the call didn’t raise an exception, which is a far weaker claim than “this endpoint works correctly.”

A test that actually defends the behavior looks different:

def test_get_user_profile():
    response = client.get("/api/users/42")
    assert response.status_code == 200
    data = response.json()
    assert data["id"] == 42
    assert data["email"] == "fixture@example.com"

Now deleting or breaking the endpoint makes the test fail. The first version doesn’t. AI assistants default to the weaker form because assert response is not None is syntactically plausible and matches the shape of what tests look like. The tool that generated the test didn’t understand what “verified” means.

The Empty Test Body

BrassCoders’s Pylint integration flags W0613 (unused argument) in some test scaffolding, which can surface functions where fixtures are set up but never used. For the specific case of a test function that sets up mocks and asserts nothing at all, there’s no Pylint rule that reliably catches it — pytest doesn’t require assertions, and the function signature gives no hint.

The pattern looks like this:

def test_send_notification(mock_email_client, mock_user_repo):
    mock_user_repo.get.return_value = User(id=1, email="test@example.com")
    mock_email_client.send.return_value = {"status": "queued"}
    # no assertion

Pytest runs that function, the mocks return their configured values, the function exits with no error, and the test passes. It’s listed as passing in your CI report. What it actually proved: nothing. The send_notification function could raise an exception, silently discard the email, or return the wrong status — the test doesn’t check any of it.

This happens because AI assistants scaffold the test correctly up through the mock setup, then sometimes stop. The assertion step requires understanding what “correct behavior” means for the code under test. When the assistant isn’t sure, it omits the assertion rather than writing a wrong one.

A pytest plugin, pytest-warn-missing-assertions, can flag test functions that exit without asserting. It won’t catch every case, but it catches the clearest: a function body that never calls assert, assertEqual, or any matcher. Worth adding to your pyproject.toml if this pattern shows up in your codebase.

Mock-Everything Tests

BrassCoders doesn’t have a static rule for mock overuse. There’s no threshold that separates “appropriate isolation” from “testing the mock, not the code,” and building one would require the kind of semantic context inference that belongs to the AI consumer of the scan output — not the scanner itself.

The pattern is worth recognizing anyway:

@patch("myapp.services.notifications.UserRepository")
@patch("myapp.services.notifications.EmailClient")
@patch("myapp.services.notifications.send_notification")
def test_send_notification(mock_send, mock_email, mock_repo):
    mock_send.return_value = {"status": "sent"}
    result = send_notification(user_id=1, message="Hello")
    assert result["status"] == "sent"

The function being tested — send_notification — is itself mocked. The test calls the mock, checks the mock’s return value, and reports success. Delete send_notification entirely, and this test still passes. That’s the diagnostic: if deleting the function under test doesn’t break the test, the test doesn’t test the function.

This pattern comes from AI assistants over-applying a learned habit. Mocking external dependencies is correct. Mocking the unit under test is a category error. The assistant sees “mock external dependencies” and applies it indiscriminately.

The fix is to check what layer each mock targets. If any mock replaces the function your test is supposed to be exercising, remove that mock and let the real function run. The external dependencies it calls can stay mocked.

What BrassCoders Catches (And What It Doesn’t)

BrassCoders is primarily a security and correctness scanner — Bandit, Pyre/Pysa taint analysis, detect-secrets, Semgrep, and the custom AI-pattern detector all focus on production code paths. Test-quality issues are a secondary concern, and the coverage is partial.

Here’s what BrassCoders does surface in test code:

Always-pass assertion patterns. Pylint’s assertion checks flag assert True, assert 1 == 1, and tautological comparisons. It also catches assert statements used as if they were function calls (no parentheses), which silently do nothing in Python.

Credentials in test fixtures. If a test file contains a hardcoded API key, AWS access key, or GitHub PAT — even in a fixture marked as fake — detect-secrets flags it. BrassCoders redacts these from the YAML output before they leave your machine. Real credentials in tests are a common CI/CD supply chain exposure point.

Phantom API calls in test utilities. The AI-pattern scanner looks for calls to functions that don’t exist in imported modules. This catches hallucinated test helper methods that AI assistants sometimes invent.

Here’s what BrassCoders doesn’t catch:

Semantic always-true assertions. assert response is not None is syntactically valid and Pylint doesn’t know the return type. Only a test that actually runs and checks the return value can tell you this assertion is weak.

Mock overuse. There’s no static rule for “this test mocks the thing it’s supposed to test.” Detecting it requires understanding the test’s intent relative to the code under test.

Missing assertions. A test function with only mock setup and no assert looks structurally valid to a linter. Pylint doesn’t flag it.

For those gaps, two tools complement BrassCoders well. mutmut (mutation testing for Python) and cosmic-ray both introduce small deliberate changes to your production code and rerun your test suite. If tests pass after a mutation, that behavior isn’t defended. Running pytest --cov --branch alongside mutation testing gives you a combined picture: which lines ran and which behavioral changes your tests would actually catch.

BrassCoders and mutation testing answer different questions. A security issue in your send_notification function is something BrassCoders will find in the production code whether or not the test suite is any good. Whether a regression in that function would be caught by a future test run is what mutation testing answers. They don’t overlap.

---

Install the OSS core free: pip install brasscoders, then brasscoders scan /path/to/project. It takes about 90 seconds on a typical Django app and emits a YAML file your AI assistant can read directly.

Full install instructions at coppersun.dev/install/.