Workflow · last verified 2026-07-13

Reviewing AI-written code: the diff discipline that keeps agents honest

The job this workflow does

The volume claims are vendor- and self-reported, but they all point the same way: Sundar Pichai said over 25% of new code at Google is AI-generated (Oct 2024), Satya Nadella put Microsoft at 20–30% (Apr 2025), and Anthropic says roughly 80% of its own production code is written by Claude as of mid-2026 — Claude Code itself reportedly around 90%. Whatever the true numbers, the review queue is now mostly machine-written diffs, and the failure data says they need review: Veracode's 2025 GenAI Code Security Report found AI-generated code introduced security flaws in 45% of 80 curated tasks across 100+ LLMs (Java worst at 72%; XSS-class tasks secure only ~12–13%), and GitClear's 2025 analysis of 211M changed lines found duplicated blocks up 8x in 2024, with copy/paste lines rising from 8.3% to 12.3% between 2021 and 2024 while refactored/moved code fell from ~25% to under 10% — the first time copy/paste exceeded moved code.

DORA 2025 names the tension precisely: higher AI adoption correlates with increased throughput and increased delivery instability — "AI amplifies" whatever process it lands in. Review is the new constraint, 30% of developers report little or no trust in AI-generated code, and the hours saved generating get re-allocated to verification. This workflow is what that verification looks like when it's a discipline rather than a mood: eight steps, each producing an artifact, tuned to the specific ways AI diffs fail — hallucinated correctness, reuse blindness, weakened gates — rather than the ways human diffs fail.

It prescribes one good way. We write it Claude-centric because that is what we run — this repo gates its own merges with fresh-context reviewer subagents and adversarial verification — but every step names the tool-agnostic principle, and step 4 points at the equivalents.

The workflow

1. Shrink the unit of review

Everything downstream is easier or impossible depending on diff size, and for AI-written code the data is stark. Anthropic's internal PR analysis (self-reported): 84% of 1,000+-line PRs get review findings, averaging 7.5 per PR, versus 31% of sub-50-line PRs averaging 0.5. Big AI diffs aren't just harder to review — they demonstrably contain more that needs finding.

So the first review act happens before the PR exists: bound the task. One purpose per PR, stated in one sentence. GitHub's own guidance for agent PRs draws the send-it-back line, and we adopt it verbatim: request a smaller PR when the diff touches more than ~5 unrelated files, when the purpose doesn't fit one sentence, when there's no plan or task description attached, or when CI is failing and the only changes are to tests. An agent will happily regenerate the work as three PRs; a human reviewer cannot happily review the one PR that should have been three.

# the reviewer's first command — size before content
gh pr diff 123 --name-only | wc -l     # file count; >5 unrelated → send back
gh pr view 123 --json title,body       # purpose in one sentence, plan attached?

2. Read the CI diff first

Before the source, read every change under .github/workflows/, the test files, and coverage or lint configuration. An agent that can't make a test pass will sometimes make the test stop running — skipped suites, loosened assertions, deleted coverage thresholds, a continue-on-error that wasn't there yesterday. GitHub's agent-PR guidance is blunt and so are we: weakened tests, coverage, or CI gates are a blocker, full stop. Not a nit, not a discussion — the PR does not proceed until the gate is restored.

# gate files first, always
git diff main...HEAD -- .github/workflows/ '*.test.*' vitest.config.* .eslintrc*
# any deletion or skip here ends the review until explained

This is two minutes of reading that neutralizes the single worst failure mode: a green checkmark that no longer means what it meant last week.

3. Demand evidence in the PR, not assertions

Anthropic's best-practices guidance for Claude Code says it directly: have Claude show evidence rather than asserting success — "the test output, the command it ran and what it returned, or a screenshot." Encode that as a PR-template requirement so it isn't renegotiated per PR. The strongest form of evidence, from GitHub's agent-PR guidance: require a test that fails on the pre-change behavior. A bug fix without a failing-then-passing test is a claim; with one, it's a demonstration.

# .github/pull_request_template.md — the evidence contract
## Evidence
- [ ] Test that FAILS on pre-change behavior (link the red run or paste output)
- [ ] Command run + verbatim output for the primary claim
- [ ] Screenshot for any UI change
## Scope
- [ ] One-sentence purpose (above) covers every file in the diff

The same principle scales down into the agent session itself: Anthropic's docs describe a verification ladder — an in-prompt check, then a /goal condition, then a deterministic Stop-hook gate, then a verification subagent that tries to refute the result. Steps 6 and 7 use the top two rungs; the point here is that "it works" is never accepted as a sentence, only as output.

4. Run the machine pass first, human pass second

Reviewer attention is the scarce resource, so spend the cheap pass first. We run Claude Code's /code-review on every branch before a human looks: it reviews the branch's commits ahead of upstream plus uncommitted changes, finds correctness bugs and reuse/simplification/efficiency issues, and can target a file, PR, branch, or ref range. Effort levels are a real dial — low/medium return fewer, high-confidence findings; high and max go broader. --comment posts findings as inline PR comments; --fix applies them to the working tree. (It was renamed from /simplify at v2.1.147, if you're looking for it under the old name.)

# machine pass, medium effort, findings as PR comments
claude /code-review 123 --comment
# heavier variants when the diff warrants it:
claude /code-review ultra          # fleet of reviewers in a remote sandbox;
                                   # every finding independently reproduced, 5–10 min
claude ultrareview --json          # CI mode: exits 0/1 on findings

Two heavier tiers exist when a diff deserves them. Ultrareview (research preview since v2.1.86) runs a fleet of reviewer agents in a remote sandbox where "every reported finding is independently reproduced and verified" — 5–10 minutes, three free runs on Pro/Max then roughly $5–20 per review, with a claude ultrareview CI mode that exits 0/1 and emits --json; note it's not available on Bedrock/Vertex or under ZDR. And Anthropic's managed Code Review GitHub service (launched 2026-03-09, Team/Enterprise research preview, ~$15–25/review) runs multi-agent parallel analysis with a verification step that filters false positives, labels findings Important / Nit / Pre-existing, triggers on @claude review, and — deliberately — never blocks merge: it posts a neutral check and exposes bughunter-severity JSON so you decide the gate (step 7). Its REVIEW.md file is injected at highest priority and is where you redefine severities, cap nits, and set the verification bar — e.g. "behavior claims need a file:line citation." CLAUDE.md violations surface as nits.

Anthropic's internal numbers on this pipeline (self-reported, so attributed as such): before, 16% of PRs got substantive review comments; with the machine pass, 54% — and under 1% of findings are marked incorrect. If your stack is elsewhere, the same slot is filled by GitHub Copilot code review (agentic, severity-labeled, analysis-depth tiers), CodeRabbit (the most-installed review app, diff-level annotations), Greptile (full-codebase context, free tier since 2026-06), or Cursor Bugbot ($40/user/mo with Autofix cloud agents) — the workflow doesn't change, only the command. The human pass then reads what survived the machine pass, which is step 5.

5. Trace one critical path by hand

The human pass is not a line-by-line read — the machine already did that. It is a single end-to-end trace of the diff's most critical path, because the machine passes are weakest exactly where AI-written code fails most convincingly: hallucinated correctness. Code that reads right, compiles, and is wrong. GitHub's reviewer checklist for AI-generated code names the concrete checks: do the imports resolve, do the called methods actually exist, do the mocks test behavior or merely test the wiring. Pick the one path where a bug costs the most — the auth check, the money movement, the data deletion — and walk it: entry point to side effect, boundary conditions at each hop, permission checks where trust changes hands.

While tracing, watch for reuse blindness: agents reimplement utilities that already exist three files away — the per-diff face of GitClear's duplication curve. A reimplemented helper isn't a style nit; it's a second copy that will drift from the first. The fix is a comment pointing at the existing util, and (step 8) a rule so the class of finding dies.

# the trace, made mechanical — for each hop on the critical path:
# 1. does this import/function exist?   git grep -n "functionName"
# 2. boundary: what happens at 0, empty, max, unauthorized?
# 3. is there already a util for this?  git grep -n "similarVerb" src/lib/

6. Split writer and reviewer contexts

The agent that wrote the code cannot grade it — not as an ethics rule but as a mechanics one. Anthropic's best-practices docs state the reason: "a fresh context improves code review since Claude won't be biased toward code it just wrote." The same context window that produced the diff contains the reasoning that justified it; asked to review, the model re-reads its own rationale and agrees with itself. So the reviewer is always a fresh context — a subagent or a new session — and it is framed adversarially: the docs describe having "a fresh model try to refute the result, so the agent doing the work isn't the one grading it."

Two framing rules make the fresh reviewer useful instead of noisy. First, the refute framing: the task is "find where this claim is false," not "review this code." Second, the over-reporting guard, straight from the docs: "tell the reviewer to flag only gaps that affect correctness" — an unconstrained reviewer pads its report with style findings to look thorough, and the real finding drowns.

# .claude/agents/adversarial-reviewer.md
---
name: adversarial-reviewer
description: Fresh-context reviewer. Tries to refute the diff's claim.
---
You did not write this code. The author claims: {CLAIM}.
Your job is to refute it. Trace the critical path; check boundaries,
permissions, and that called symbols exist. Flag ONLY gaps that affect
correctness — no style, no preferences. Every finding needs file:line.

This is first-person practice, not theory: this repo's merges are gated by exactly this pattern — fresh-context reviewer subagents plus adversarial verification — and the findings that survive the "refute it" framing are reliably the ones worth a human's time.

7. Gate deterministically — advisory rules don't gate

Everything so far produces findings; something has to refuse the merge. Anthropic's docs draw the distinction cleanly: hooks are deterministic, CLAUDE.md is advisory. A rule the model is asked to follow is a suggestion; a gate that runs as code is a gate. Inside the agent session, that's a Stop hook — a deterministic check (tests, lint, typecheck) that blocks the agent from declaring itself done until the check passes (it can be overridden only after 8 consecutive blocks, which is the escape valve, not the norm). In CI, it's an exit code: claude ultrareview --json exits 0/1, and the managed Code Review service's bughunter-severity JSON exists precisely so you can fail the build on Important findings even though the service itself never blocks merge.

# .claude/settings.json — the agent can't stop until the suite is green
{ "hooks": { "Stop": [ { "hooks": [ { "type": "command",
      "command": "npm run typecheck && npm test" } ] } ] } }

# CI — the org decides the gate, not the review service
claude ultrareview --json | jq -e '[.findings[] | select(.severity=="important")] | length == 0'

The principle generalizes past any one tool: every rule you actually rely on must exist as something that can return a nonzero exit code. The trust-then-verify gap is a named failure mode in Anthropic's own guidance — "if you can't verify it, don't ship it" — and a gate that lives only in prose is a gap wearing a rule's clothes.

8. Feed findings back so the class of bug dies

A finding fixed once is a cost; a finding converted into a rule is an asset. When the same class of issue shows up in a second review, it stops being a review comment and becomes: a CLAUDE.md rule (advisory — shapes what the agent writes next time; violations surface as nits in the managed service), a REVIEW.md rule (shapes what the reviewer hunts for and the evidence bar it enforces), or — best of all — a lint rule or test, which is deterministic and free forever.

# REVIEW.md — injected highest-priority into the managed review
## Severity overrides
- Any change under .github/workflows/ that removes a step: Important, always.
- Reimplementing a helper that exists in src/lib/: Important, cite the existing file.
## Verification bar
- Behavior claims need a file:line citation.
- Max 3 nits per review; prefer zero.

Route by determinism: if a machine can check it, make it lint or a test; if only a model can check it, put it in REVIEW.md; if it's about how code should be written in the first place, CLAUDE.md. The measure of a review process is not findings per PR — it's whether last quarter's most common finding is extinct.

Artifacts this workflow produces

• REVIEW.md — severity redefinitions, nit caps, and the verification bar, injected highest-priority into machine review.
• The reviewer-subagent definition — fresh-context, adversarial refute framing, correctness-only scope, file:line citations required.
• The Stop-hook gate — deterministic typecheck+test gate the writing agent cannot talk its way past.
• The PR template — evidence contract: failing-test-first, command output, one-sentence scope.
• The severity-JSON CI gate — the pipeline step that fails the build on Important findings, since the review services deliberately never block merge themselves.

Pitfalls

• Reviewing the source before the gates. A weakened workflow file makes every subsequent green check a lie. CI, test, and coverage diffs first, always.
• Letting the writer grade itself. Same context = same bias; the model re-reads its own rationale and approves it. Fresh context or it isn't a review.
• Accepting assertions as evidence. "Tests pass" is a sentence. The test output, the command and its return, or the screenshot — or it didn't happen.
• Unconstrained reviewers. A reviewer not told "correctness gaps only" pads with style findings, and the one real bug drowns in nits.
• Advisory rules doing a gate's job. CLAUDE.md shapes behavior; it does not refuse merges. Anything you rely on must exit nonzero somewhere.
• Rubber-stamping small-looking diffs from big prompts. Diff size is a proxy, not an absolution — a 30-line auth change still gets the hand trace (step 5).
• Treating reuse blindness as style. The duplicate helper is tomorrow's divergent behavior. It's a correctness finding with a delay on it.
• Fixing findings without killing the class. If the same finding appears twice and there's no new rule or lint, the review process is a treadmill, not a ratchet.

When not to use this workflow

Don't run the full eight steps on throwaway code: spikes, one-off analysis scripts, prototypes that will be deleted rather than merged. The discipline exists to protect a trunk that ships; code that never merges needs only step 5's instinct — don't act on its output without a sanity check.

And don't use review as a substitute for specification. If every PR from an agent needs heavy correction, the defect is upstream — in the task framing, the CLAUDE.md, or the missing eval harness. That's the territory of eval-first development and the Claude Code workflow: get the writing loop right and this workflow gets cheaper, because review is the backstop, not the process.

Changelog

• 2026-07-13 — initial version, verified against Claude Code /code-review docs and 2025–26 industry studies.