Debugging with AI

This lesson on Debugging with AI is hands-on and example-driven. You will isolate software bugs into minimal reproducible tests and direct LLM assistants through targeted root-cause analysis rather than speculative multi-file editing. You will enforce safe reversion checkpoints and small context windows to secure verified fixes in a single iteration round.

What You'll Be Able To Do

• Construct an automated failing test case that reliably reproduces the bug in isolation.
• Establish a version control checkpoint containing the failing test before engaging the AI assistant.
• Formulate root-cause diagnostic prompts that request causal explanations instead of blind code patches.
• Evaluate AI diagnostic output for failure signals including recycled proposals, wandering edits, and high conversational turns.
• Execute full-suite regression tests and commit minimal targeted patches to the repository.

Detailed Concept Walkthrough

1. Failure States and Core AI Limitations

AI coding assistants operate on probabilistic token completion rather than execution logic, rendering high confidence uncoupled from factual correctness. Recognizing degradation patterns prevents costly patch accumulation loops.

• Under the Hood: Large language models calculate the most probable textual response based on training data patterns without executing runtime code or validating environmental state.
• Failure Signals: Three definitive triggers indicate session decay: cycling through identical or recycled proposals, proposing wandering edits to unrelated files, and exceeding five conversational rounds.
• The Ablation Loop: When interacting repeatedly without rollbacks, the model begins debugging the side effects of its own previously introduced faulty patches instead of the underlying original bug.

# Identify failure state symptoms early:
# 1. AI edits config/unrelated files when debugging a logic bug.
# 2. AI repeats a patch you already rejected.
# 3. Conversation exceeds 5 turns.
# Action: Terminate session immediately, revert to base commit.

Key Takeaway: AI textual confidence is completely independent of functional correctness, requiring strict turn limits and immediate rollbacks on conversational decay.

2. The Save Point and Reproduction Workflow

An explicit failing test traps the bug under controlled conditions, and a dedicated Git commit creates a safe baseline to prevent cascading code corruption.

• Mechanism: Building a deterministic, failing test case (the 'trap') creates an objective, automated binary gate that proves whether the bug exists before and after patching.
• Execution Flow: Create the failing test script, run it to verify the red state, and immediately execute a version control commit to record the baseline state before prompting the AI.
• Best Practice: Keep the reproduction script completely self-contained so it executes fast and pinpoints the exact boundary where expected and actual behaviors diverge.

# 1. Write the isolated failing test
pytest tests/test_payment_calculator.py::test_discount_application

# 2. Commit the failure state to establish a clean rollback point
git add tests/test_payment_calculator.py
git commit -m "test: capture discount calculation failure state"

Key Takeaway: Never prompt an AI for a code fix without a committed failing test that objectively verifies the problem.

3. Context Optimization and Isolation

Supplying excessive codebase context degrades LLM output accuracy by introducing noise, making minimal file injection essential for reliable results.

• The Context Paradox: Supplying entire repositories (hundreds of files) degrades accuracy because the model relies on surface-level keyword matching and broad generalizations instead of deep logic tracing.
• Mechanism: Isolate the prompt context strictly to the red test, expected outcomes, and no more than three tightly coupled target source files in a fresh conversation.
• Best Practice: Always start a new chat session when initiating a fix to clear out hallucinations, stale assumptions, and legacy conversational context.

# Context boundary: Supply strictly <= 3 files to the new AI chat
# 1. Failing test: tests/test_payment_calculator.py
# 2. Implementation: src/payment_calculator.py
# 3. Shared types/interface: src/models/discount.py

Key Takeaway: Constrain AI context to three files or fewer inside a clean chat session to prevent conversational bias and hallucinated fixes.

4. Root-Cause Diagnosis and Verification

Forcing the model to explain the failure mechanism prior to generating code ensures that the patch addresses root causes rather than masking symptoms.

• Mechanism: Prompting the AI to list potential technical causes and explain the mechanics behind the test failure forces structural analysis over shallow patch generation.
• Execution Flow: Validate the AI's explanation against observable system symptoms and check whether the proposed theory correctly predicts additional secondary behaviors.
• Validation Rule: Once a patch is applied, execute the specific unit test until green, run the entire regression test suite, and verify the patch is minimal and clean before merging.

# Prompt Pattern for Forensic Analysis:
"""
Explain why the test `test_discount_application` fails when given the implementation
in `payment_calculator.py`. List the top 3 potential technical root causes before 
suggesting any code modifications.
"""

Key Takeaway: Demand a causal explanation before requesting code changes, then validate the fix against both the trap test and the full test suite.

Topics Covered in Debugging with AI

1. Cost of Blind Debugging (00:00 - 02:02) — Unstructured prompts and insufficient context lead to rapid patch accumulation and unresolved bugs.
2. AI Failure States (02:02 - 02:47) — Recognizable warning signs include recycled proposals, wandering edits, and exceeding five conversational rounds.
3. Core AI Limitations (02:47 - 04:16) — Language models generate pattern-based hypotheses without runtime execution, divorcing linguistic confidence from factual correctness.
4. Reproducible Failure State (04:16 - 05:03) — Creating an automated failing test isolates the bug and provides an objective benchmark for verifying fixes.
5. Narrowing Search Space (05:03 - 06:06) — Using the assistant for forensic hypothesis generation pinpoints underlying technical causes before code editing begins.
6. Safe Reversion Points (06:06 - 06:38) — Committing the failing test in version control establishes a clean rollback point that prevents cascading errors.
7. Context Optimization (06:38 - 07:34) — Restricting prompt input to three files or fewer inside a fresh session prevents contextual degradation.
8. Explanation Over Modification (07:34 - 08:28) — Prompting for causal root-cause explanations verifies the problem mechanism before generating code patches.
9. Validation and Prevention (08:28 - 09:07) — Running the trap test and the full test suite confirms the patch while preventing future regressions.

AI-Assisted Coding Cheat Sheet

• git commit -m "test: reproduce bug" — Creates safe rollback point before asking AI for fixes

  git add tests/test_bug.py && git commit -m "test: trap bug"
  

• git reset --hard HEAD — Reverts failed AI edits back to baseline test state

  git reset --hard HEAD
  

• pytest <path/to/test.py> — Executes trap test to verify red or green state

  pytest tests/test_auth.py -k test_token_expiration
  

• git diff --stat — Audits AI patch size to ensure minimal file modifications

  git diff --stat HEAD~1
  

• pytest — Runs full test suite to guard against regressions

  pytest --maxfail=1
  

Comparison Table

Common Pitfalls

• Mistake: Requesting fixes across many conversational rounds. Avoid: Reset to the base commit and start a clean chat when reaching five rounds or seeing wandering edits.
• Mistake: Providing hundreds of repository files to the prompt. Avoid: Limit context strictly to the failing test, expected outcome, and maximum three target source files.
• Mistake: Prompting the AI to fix code before diagnosing. Avoid: Demand a technical explanation of why the test fails and validate the reasoning before accepting code.
• Mistake: Applying AI patches without version control checkpoints. Avoid: Commit the failing test case before prompting the AI so broken changes can be instantly reverted.

FAQs

• Why does an AI assistant start proposing edits to completely unrelated files? The model has lost the conversational thread and is cycling through generic training patterns rather than analyzing the specific logic path.
• What should I do when an AI suggests a fix that sounds highly confident but fails? Discard the patch immediately with Git reset; the model's linguistic confidence reflects probabilistic phrasing, not functional accuracy.
• Should the temporary failing test created during debugging remain in the codebase? Yes, keep the passing test committed in the repository to serve as permanent regression prevention against future code changes.