Build 5 — AI Code Review of a Real PR

🎯 Hands-on Portfolio Deliverable: Perform an end-to-end comparative code review on a pull request using AI, calibrating AI reviewer capabilities against human review, and establishing a pre-PR self-review protocol.

The brief

Run AI code review on a real PR. Compare findings to human reviewers (or to your own review). Build calibration on what AI catches vs misses.

Pick a PR

Options:

Your own open PR (best — you understand the code).
A colleague's PR you're reviewing.
A recent PR in an open-source project (look at Issues for "good first review" or similar).

Step 1: Read the PR yourself first (15 min)

DON'T use AI yet. Read the diff. Note issues you spot, things you'd ask about.

# My Review

- [must-fix] X function doesn't handle empty input.
- [suggest] Y could use a clearer name.
- [nit] Inconsistent quote style.

Step 2: AI review (15 min)

Now have AI review:

"Review this PR. Focus on:

Correctness (logic bugs, edge cases).

Security (input validation, secrets, auth).

Code quality (naming, clarity, complexity).

Test coverage.

Style (matches conventions).

Performance, where relevant to this change.

Tag as [must-fix] / [suggest] / [nit]."

Paste the diff or, in Cursor / Claude Code, just say "review this branch".

Step 3: Compare reviews (15 min)

Compare up to six distinct findings or checks. If the PR has fewer than six meaningful items, document the checks performed and explain; do not invent defects.

| Issue | You | AI | Real? |
|-------|-----|-----|------|
| Empty input not handled | ✓ | ✓ | yes |
| Off-by-one in pagination | ✗ | ✓ | yes |
| Variable name unclear | ✓ | ✓ | yes |
| Missing test for error case | ✗ | ✓ | yes |
| "Use list comprehension here" | ✗ | ✓ | nit; not really |
| "Add docstring" | ✗ | ✓ | nit; not really |
| Subtle race condition | ✓ | ✗ | yes — AI missed |
| Wrong assumption about input data | ✓ | ✗ | yes — domain knowledge |

You'll see:

AI catches different things than you.
AI catches some real things you missed (esp. mechanical bugs).
AI misses some things you caught (esp. domain-specific).
AI is over-eager on nits.

Step 4: Decide which feedback to apply

Filter AI feedback:

Apply must-fix items.
Consider suggests.
Skip nits unless legitimately useful.

Step 5: Reflect

# AI Review Reflection

## PR: [link or description]

## What AI caught that I missed:
- ...

## What I caught that AI missed:
- ...

## False positives from AI:
- ...

## When I'd use AI review going forward:
- ...

## When I'd skip it:
- ...

Patterns in AI review

After doing this on a few PRs, you'll notice:

AI is GOOD at

Pattern-matched bugs (string concat in SQL, etc.).
Missing error handling.
Style inconsistencies.
Untested edge cases.
Outdated patterns.

AI is BAD at

Domain logic correctness.
Subtle race conditions.
Architectural fit.
Team-specific conventions (without CLAUDE.md).
Knowing when a "violation" is actually intentional.

Calibrate trust accordingly.

Pre-PR self-review with AI

For your own PRs, do this BEFORE opening:

1. Make changes.
2. Self-review.
3. AI review.
4. Address AI's must-fixes.
5. Open PR with humans.

Saves human-reviewer time on issues AI catches.

Comparing across PRs

Try this on 3-5 PRs of varying complexity:

Small bug fix.
Medium feature.
Large refactor.

Track: AI's hit rate, miss rate, false-positive rate.

You'll find AI scales differently across PR types.

Deliverables

AI review output for at least one real PR.
Comparison table (yours vs AI).
Reflection on when AI helps vs not.

What you'll learn

Calibration: AI as reviewer has predictable strengths and gaps.
AI catches mechanical things you miss.
You catch domain things AI misses.
AI nits should be filtered.
The pre-PR self-review pattern.

Stretch

Set up AI review in CI (some tools support auto-review).
Apply this to a security-focused review specifically.
Try on a 100+ file refactor; assess if AI keeps up.

📊 100-Point Evaluation Rubric

Evaluation Dimension	Weight	Target Standard (Pass ≥ 80%)
1. Independent Human Review	25 pts	Thorough manual code review documented prior to AI generation, identifying domain-specific nuances and logic requirements.
2. Prompted AI Review Quality	25 pts	AI review executed across correctness, security, performance where relevant, test coverage, and style, tagged with severity levels (`[must-fix]`, `[suggest]`, `[nit]`).
3. Calibration Matrix	25 pts	Detailed comparison table categorizing findings (You vs AI), identifying real defects caught by AI vs false alarms/nits and missed domain logic.
4. Protocol & Reflection	25 pts	Actionable pre-PR self-review checklist formulated for ongoing engineering use, supported by thoughtful reflection on AI review trust boundaries.

📤 Submission Instructions & Guidelines

Deliverable Formats:
- Option A (Recommended): Upload your CODE_REVIEW_ANALYSIS.md (or .zip containing the analysis report and PR diff).
- Option B: Upload your markdown report and paste the link to the reviewed PR in the Submission Notes box below.
Pre-Submission Verification:
- Compare up to six genuine findings or checks; if fewer apply, explain the checks performed without inventing defects.
- Confirm clear distinctions between must-fix issues and cosmetic nits.
Submit Your Work:
- Upload your review document below and click Submit Assignment.

Assignment Instructions