Testing AI-Generated Code: 5 Checks QA Teams Miss

A developer asks an AI assistant to build a refund feature. The requirement says refunds are allowed within 30 days, and total refunds on an order can never exceed what the customer paid. The assistant writes the function. The developer asks it to write the tests too. Everything passes, the pull request looks clean, and the feature ships.

Two weeks later, finance notices customers receiving more money back than they paid. A customer can request a $60 refund, then another $50 on a $100 order, because the code only compares each request to the order total and never to what was already refunded.

The tests did not catch it because they were written from the code, not from the requirement. Both artifacts shared the same blind spot. This is the pattern QA teams are running into as AI-generated code moves into production, and it is why a green build now means less than it used to.

Why AI-generated code fails differently

Human mistakes tend to look like mistakes. A typo, a missing null check, a forgotten edge case. AI-generated code is usually fluent and well structured, which makes problems harder to see in review.

DeviQA's methodology for AI-assisted projects describes the failure modes well: generated code can be syntactically correct while carrying incorrect assumptions, incomplete business logic, excessive scope or hidden dependencies, and AI-written tests are not accepted as sufficient proof that the implementation is right.

In practice, the bugs that slip through are rarely crashes. They are plausible behaviors that nobody asked for, or rules that quietly disappeared between the requirement and the prompt. Traditional regression suites were built for a world where developers read the requirement themselves. Here are five checks that fit this new reality.

Check 1: Write tests from the requirement before reading the code

This is the most important check and the one most often skipped. If the same AI writes the code and the tests, the tests inherit its interpretation. A passing result then confirms that the code matches itself.

One 2026 QA article makes the same point, arguing that writing tests from the requirement first removes the circular dependency between AI-generated code and AI-generated tests.

For the refund example, a test written from the requirement looks like this:

Order total: $100. Already refunded: $60.
Request: refund $50.
Expected: rejected, because total refunds would be $110.

The AI-written test, derived from the code, checked that $50 is less than or equal to $100. It passed. The requirement-first test fails immediately.

This only works if the requirement is precise enough to test. If yours are vague, start with how to write testable requirements. Many teams find that the exercise of writing the test exposes the gap in the requirement before any code exists.

Check 2: Compare what the code does against what was asked

AI tools often do more than requested. A refund function might also send an email, write an audit record, or add a retry loop. Extra behavior is untested behavior, and it can create compliance or security exposure.

Review in both directions. For each requirement, confirm there is code and a test. For each significant piece of code, confirm there is a requirement. A traceability matrix makes this visible, and it is also what auditors in banking and healthcare ask for. See requirements traceability matrix automation for how teams keep this current without a spreadsheet nobody maintains.

A frequent issue we see is a requirement that changed during development while the prompt did not. The AI built the old version, and nobody noticed because the tests were generated from the same stale prompt.

Check 3: Run an adversarial edge case pass with a separate prompt

The first prompt asks the AI to build. A second, separate prompt should ask it to break what was built. Give it the requirement and the implementation, then ask for inputs and sequences that would violate the requirement.

For the refund feature, a good adversarial pass produces cases like these:

• Two refund requests submitted at the same moment
• A refund on the 30th day at 11:59 pm in a different time zone
• A partial refund followed by a full refund request
• A negative or zero amount sent directly to the API
• A refund against an order that was already cancelled

Keep a human in the loop for this step. The model tends to propose cases it can already handle. The edge cases that matter most come from knowing how your business actually behaves. The NATS 2023 flight plan failure is a good reminder that one rare input can take down a system that passed every ordinary test.

Check 4: Have a person decide what correct means

When an AI writes both the code and the expected results, it also decides what a pass looks like. That is a risk in any domain with money, safety or regulation attached. A tool can check whether output matches an expectation. It cannot tell you whether the expectation is right.

Assign a named person to confirm expected results for high-risk behavior, such as calculations, eligibility rules and access control. Record who approved them. This is the idea behind a test oracle, and it becomes more important, not less, as more of the pipeline is automated. Teams building governance around this will also find AI governance in software testing useful.

Check 5: Review security and dependencies separately

AI-generated code touching authentication, payments or personal data deserves its own review, in addition to normal security review. Fluent code can look more trustworthy than it is.

One specific issue is package hallucination. A study by researchers at the University of Texas at San Antonio, Virginia Tech and the University of Oklahoma analyzed more than 576,000 code samples from 16 models and found that at least 5.2% of packages suggested by commercial models and 21.7% from open-source models were hallucinated.

The models tested were a snapshot in time, so rates today will differ, but the risk is real. Attackers can register a fake package under a name an AI keeps inventing, and a developer who installs it without checking pulls malicious code into the build.

Practical steps:

• Verify every new dependency exists, is maintained and is the package you meant
• Pin versions and use lockfiles
• Run dependency and vulnerability scanning in CI
• Check generated code against the OWASP Top 10 for injection, broken access control and insecure handling of input

Put more scrutiny where the risk is

You cannot give every AI-written change the same level of review. A risk gate helps.

Payments, refunds, pricing

• Requirement-first testing
• Human-approved expected results
• Adversarial testing

Authentication, permissions

• Requirement-first testing
• Human-approved expected results
• Adversarial testing
• Separate security review

Personal or health data

• Requirement-first testing
• Human-approved expected results
• Adversarial testing
• Compliance sign-off

Internal tooling, UI copy

• Standard review
• Automated regression testing

This follows the logic of risk-based testing. The aim is not to slow everything down. It is to spend careful attention where a mistake costs the most. The history of failures like Knight Capital shows what happens when high-risk changes receive routine checks.

• testing ai-generated code
• qa automation
• requirements based testing