Build 5 — AI Code Review of a Real PR

# Build 5 — AI Code Review of a Real PR ## The brief Run AI code review on a real PR. Compare findings to human reviewers (or to your own review). Build calibration on what AI catches vs misses. ## Pick a PR Options: 1. Your own open PR (best — you understand the code). 2. A colleague's PR you're reviewing. 3. A recent PR in an open-source project (look at Issues for "good first review" or similar). ## Step 1: Read the PR yourself first (15 min) DON'T use AI yet. Read the diff. Note issues you spot, things you'd ask about. ```markdown # My Review - [must-fix] X function doesn't handle empty input. - [suggest] Y could use a clearer name. - [nit] Inconsistent quote style. ``` ## Step 2: AI review (15 min) Now have AI review: > "Review this PR. Focus on: > 1. Correctness (logic bugs, edge cases). > 2. Security (input validation, secrets, auth). > 3. Code quality (naming, clarity, complexity). > 4. Test coverage. > 5. Style (matches conventions). > > Tag as [must-fix] / [suggest] / [nit]." Paste the diff or, in Cursor / Claude Code, just say "review this branch". ## Step 3: Compare reviews (15 min) ``` | Issue | You | AI | Real? | |-------|-----|-----|------| | Empty input not handled | ✓ | ✓ | yes | | Off-by-one in pagination | ✗ | ✓ | yes | | Variable name unclear | ✓ | ✓ | yes | | Missing test for error case | ✗ | ✓ | yes | | "Use list comprehension here" | ✗ | ✓ | nit; not really | | "Add docstring" | ✗ | ✓ | nit; not really | | Subtle race condition | ✓ | ✗ | yes — AI missed | | Wrong assumption about input data | ✓ | ✗ | yes — domain knowledge | ``` You'll see: - AI catches different things than you. - AI catches some real things you missed (esp. mechanical bugs). - AI misses some things you caught (esp. domain-specific). - AI is over-eager on nits. ## Step 4: Decide which feedback to apply Filter AI feedback: - Apply must-fix items. - Consider suggests. - Skip nits unless legitimately useful. ## Step 5: Reflect ```markdown # AI Review Reflection ## PR: [link or description] ## What AI caught that I missed: - ... ## What I caught that AI missed: - ... ## False positives from AI: - ... ## When I'd use AI review going forward: - ... ## When I'd skip it: - ... ``` ## Patterns in AI review After doing this on a few PRs, you'll notice: ### AI is GOOD at - Pattern-matched bugs (string concat in SQL, etc.). - Missing error handling. - Style inconsistencies. - Untested edge cases. - Outdated patterns. ### AI is BAD at - Domain logic correctness. - Subtle race conditions. - Architectural fit. - Team-specific conventions (without CLAUDE.md). - Knowing when a "violation" is actually intentional. Calibrate trust accordingly. ## Pre-PR self-review with AI For your own PRs, do this BEFORE opening: ``` 1. Make changes. 2. Self-review. 3. AI review. 4. Address AI's must-fixes. 5. Open PR with humans. ``` Saves human-reviewer time on issues AI catches. ## Comparing across PRs Try this on 3-5 PRs of varying complexity: - Small bug fix. - Medium feature. - Large refactor. Track: AI's hit rate, miss rate, false-positive rate. You'll find AI scales differently across PR types. ## Deliverables - AI review output for at least one real PR. - Comparison table (yours vs AI). - Reflection on when AI helps vs not. ## What you'll learn - Calibration: AI as reviewer has predictable strengths and gaps. - AI catches mechanical things you miss. - You catch domain things AI misses. - AI nits should be filtered. - The pre-PR self-review pattern. ## Stretch - Set up AI review in CI (some tools support auto-review). - Apply this to a security-focused review specifically. - Try on a 100+ file refactor; assess if AI keeps up.