Writing Tests with AI

This lesson on Writing Tests with AI is hands-on and example-driven. You will learn to identify and mitigate AI-specific failure modes, such as confident hallucinations, edge-case omissions, and silent contract drift. By integrating property-based testing and mutation testing into an iterative refinement loop, you will establish rigorous, machine-checkable quality gates for AI-generated code.

What You'll Be Able To Do

• Identify unique AI failure modes including confident hallucinations, common-case bias, and silent contract drift.
• Implement the four-step Generate, Review, Test, Refine feedback loop using failing test output to guide prompt iteration.
• Formulate domain invariants in plain language and translate them into property-based tests using Hypothesis.
• Configure mutation testing as an automated CI gate to verify test suite quality on files touched by AI.
• Partition testing responsibilities between human judgment for invariant design and AI generation for boilerplate suites.

Detailed Concept Walkthrough

1. Failure Profiles in AI-Generated Code

AI-generated code fails differently than human-written code by defaulting to common patterns while silently omitting subtle boundary conditions and contracts. Treating AI code as untrustworthy by default prevents silent regressions from reaching production.

• Mechanism: Large Language Models are probabilistic pattern matchers trained on typical code snippets, meaning they naturally optimize for the happy path and common cases while omitting obscure edge cases.
• Under the Hood: Silent contract drift occurs when an AI subtly alters method signatures, data shapes, or side effects without raising syntax errors, keeping downstream callers vulnerable.
• Best Practice: Require explicit, machine-checkable verification evidence before merging any AI output rather than relying on visual scan reviews.

# Example of silent contract drift: AI omitted None validation and changed return type
def parse_user_age(payload: dict) -> int:
    # Risky AI code: assumes 'age' always exists and is a valid int string
    return int(payload["age"])

# Desired hardened contract:
def parse_user_age_safe(payload: dict) -> int | None:
    val = payload.get("age")
    if val is None or not str(val).isdigit():
        return None
    return int(val)

Key Takeaway: Treat all AI-generated code as unverified by default until proven sound against explicit boundary tests.

2. The Trust-but-Verify Refinement Loop

The Generate, Review, Test, Refine workflow uses automated test failures as objective prompt feedback to correct AI implementations without manual patching.

• Execution Flow: A human defines the contract, prompts the LLM for the initial implementation, reviews the logic structure, runs automated tests, and pipes failed outputs back to the prompt.
• Mechanism: Passing raw runtime stack traces and assertion errors into the model leverages its context window to correct logic errors faster than manual human editing.
• Best Practice: Never patch AI-generated logic manually during development cycles; update the test suite to expose the bug, then prompt the AI to refine its output.

# Step 1: Write an explicit failing test exposing the missing edge case
def test_parse_user_age_missing_key():
    assert parse_user_age_safe({}) is None

# Step 2: Feed the failure output back to the LLM prompt:
# "The previous function failed test_parse_user_age_missing_key with KeyError: 'age'.
#  Refactor parse_user_age to satisfy the contract without throwing exceptions."

Key Takeaway: Drive AI code refinement through automated test failures rather than manual code surgery.

3. Property-Based Testing for Invariant Discovery

Property-Based Testing (PBT) validates system behavior across thousands of randomized inputs against high-level invariants rather than a few hardcoded examples.

• Mechanism: PBT frameworks like Hypothesis generate pseudo-random inputs, search for edge-case counterexamples, and automatically shrink failing inputs to the minimal reproduction case.
• Division of Labor: Humans author the domain invariants in plain language (e.g., 'the output string length must equal input length'), while AI translates these rules into PBT code.
• Best Practice: Use PBT specifically to stress-test numerical operations, serialization routines, and parsing logic generated by AI assistants.

from hypothesis import given, strategies as st

# Human-defined invariant: Reversing a list twice returns the original list
@given(st.lists(st.integers()))
def test_list_reversal_invariant(xs):
    assert list(reversed(list(reversed(xs)))) == xs

Key Takeaway: Define invariants in plain language and let PBT generate thousands of adversarial inputs to expose hidden AI logic flaws.

4. Mutation Testing as a Quality Gate

Mutation testing introduces intentional faults into source code to verify that the test suite actually fails, exposing weak assertions and false coverage metrics.

• Mechanism: A mutation engine creates variants ('mutants') by changing operators, conditional bounds, and return values, measuring how many mutants are 'killed' by failing tests.
• Under the Hood: High line coverage from AI-generated tests often hides weak or missing assertions; a high mutation score proves the tests actively assert correct behavior.
• Best Practice: Integrate mutation testing into CI exclusively for files modified by AI to prevent long pipeline runtimes while ensuring strict quality standards.

# Run mutation testing specifically on AI-touched modules
# mutmut generates mutants and verifies existing tests fail
mutmut run --paths-to-mutate=src/ai_generated_service.py

# Check the results summary
mutmut results

Key Takeaway: Rely on mutation score rather than line coverage to measure the real bug-catching power of AI-authored tests.

Topics Covered in Writing Tests with AI

1. Risks of AI-Generated Code (0:41 - 2:33) — Identifies failure modes such as confident errors, common-case bias, and silent contract drift.
2. The Trust-but-Verify Loop (2:33 - 3:42) — Establishes the four-step iterative cycle of Generate, Review, Test, and Refine.
3. Expanding the Testing Pyramid (3:42 - 4:28) — Introduces specification conformance, property-based testing, and mutation testing layers.
4. Guarding Against False Confidence (4:28 - 5:06) — Explains how mutation testing detects weak assertions in high-coverage test suites.
5. Implementing Property-Based Testing (5:06 - 6:24) — Demonstrates defining invariants and generating adversarial inputs using Hypothesis.
6. Strategic Division of Testing Labor (6:24 - 7:33) — Delegates test volume generation to AI while reserving invariant definition for humans.
7. Mutation Testing as a CI Gate (7:33 - 8:38) — Details how to configure mutation scores as automated CI quality gates on AI-modified code.

AI-Assisted Coding Cheat Sheet

• @given(strategies) — Generates randomized test inputs based on defined data types

  @given(st.text(), st.integers())
  def test_func(txt, num): pass
  

• mutmut run — Executes mutation testing against specified file paths

  mutmut run --paths-to-mutate=core/logic.py
  

• mutmut results — Displays summary of survived and killed code mutants

  mutmut results
  

• mutmut show <id> — Inspects specific mutant diff that survived the test suite

  mutmut show 3
  

• pytest --hypothesis-show-statistics — Shows input distributions and shrink operations for property tests

  pytest --hypothesis-show-statistics
  

Comparison Table

Common Pitfalls

• Mistake: Relying on line coverage to validate AI-generated test suites. Avoid: Use mutation testing to verify that tests actively catch seeded bugs.
• Mistake: Letting the AI define domain invariants without human oversight. Avoid: Define explicit functional invariants yourself before instructing the AI to generate property tests.
• Mistake: Manually fixing edge-case bugs in AI-generated code. Avoid: Feed machine-checkable failing test outputs back into the LLM prompt during refinement.
• Mistake: Running mutation testing across the entire codebase on every pull request. Avoid: Target mutation testing strictly on files modified by AI assistants.

FAQs

• Why is high line coverage insufficient for AI-generated code? AI assistants frequently generate tests that execute lines without asserting critical state changes, creating false confidence.
• How does property-based testing differ from parameterized testing? Parameterized tests check explicit, hardcoded examples, while property-based testing explores thousands of generated inputs to break invariants.
• What should I feed into the LLM when an AI-generated test fails? Provide the original function, the failing test case, and the exact assertion error or stack trace.
• How can we run mutation testing in CI without causing pipeline bottlenecks? Scope the mutation runner strictly to diffs and files modified by AI within that specific pull request.