Workflow · last verified 2026-07-25

Reviewing AI-written code: the diff discipline that keeps agents honest

The job this workflow does

The volume claims are vendor- and self-reported, but they all point the same way: Sundar Pichai said over 25% of new code at Google is AI-generated (Oct 2024), Satya Nadella put Microsoft at 20–30% (Apr 2025), and Anthropic says roughly 80% of its own production code is written by Claude as of mid-2026 — Claude Code itself reportedly around 90%. Whatever the true numbers, the review queue is now mostly machine-written diffs, and the failure data says they need review: Veracode's 2025 GenAI Code Security Report found AI-generated code introduced security flaws in 45% of 80 curated tasks across 100+ LLMs (Java worst at 72%; XSS-class tasks secure only ~12–13%), and GitClear's 2025 analysis of 211M changed lines found duplicated blocks up 8x in 2024, with copy/paste lines rising from 8.3% to 12.3% between 2021 and 2024 while refactored/moved code fell from ~25% to under 10% — the first time copy/paste exceeded moved code.

DORA 2025 names the tension precisely: higher AI adoption correlates with increased throughput and increased delivery instability — "AI amplifies" whatever process it lands in. Review is the new constraint, 30% of developers report little or no trust in AI-generated code, and the hours saved generating get re-allocated to verification. This workflow is what that verification looks like when it's a discipline rather than a mood: eight steps, each producing an artifact, tuned to the specific ways AI diffs fail — hallucinated correctness, reuse blindness, weakened gates — rather than the ways human diffs fail.

It prescribes one good way. We write it Claude-centric because that is what we run — this repo gates its own merges with fresh-context reviewer subagents and adversarial verification — but every step names the tool-agnostic principle, and step 4 points at the equivalents.

The workflow

1. Shrink the unit of review

Everything downstream is easier or impossible depending on diff size, and for AI-written code the data is stark. Anthropic's internal PR analysis (self-reported): 84% of 1,000+-line PRs get review findings, averaging 7.5 per PR, versus 31% of sub-50-line PRs averaging 0.5. Big AI diffs aren't just harder to review — they demonstrably contain more that needs finding.

So the first review act happens before the PR exists: bound the task. One purpose per PR, stated in one sentence. GitHub's own guidance for agent PRs draws the send-it-back line, and we adopt it verbatim: request a smaller PR when the diff touches more than ~5 unrelated files, when the purpose doesn't fit one sentence, when there's no plan or task description attached, or when CI is failing and the only changes are to tests. An agent will happily regenerate the work as three PRs; a human reviewer cannot happily review the one PR that should have been three.

# the reviewer's first command — size before content
gh pr diff 123 --name-only | wc -l     # file count; >5 unrelated → send back
gh pr view 123 --json title,body       # purpose in one sentence, plan attached?

2. Read the CI diff first

Before the source, read every change under .github/workflows/, the test files, and coverage or lint configuration. An agent that can't make a test pass will sometimes make the test stop running — skipped suites, loosened assertions, deleted coverage thresholds, a continue-on-error that wasn't there yesterday. GitHub's agent-PR guidance is blunt and so are we: weakened tests, coverage, or CI gates are a blocker, full stop. Not a nit, not a discussion — the PR does not proceed until the gate is restored.

# gate files first, always
git diff main...HEAD -- .github/workflows/ '*.test.*' vitest.config.* .eslintrc*
# any deletion or skip here ends the review until explained

This is two minutes of reading that neutralizes the single worst failure mode: a green checkmark that no longer means what it meant last week.

3. Demand evidence in the PR, not assertions

Anthropic's best-practices guidance for Claude Code says it directly: have Claude show evidence rather than asserting success — "the test output, the command it ran and what it returned, or a screenshot." Encode that as a PR-template requirement so it isn't renegotiated per PR. The strongest form of evidence, from GitHub's agent-PR guidance: require a test that fails on the pre-change behavior. A bug fix without a failing-then-passing test is a claim; with one, it's a demonstration.

# .github/pull_request_template.md — the evidence contract
## Evidence
- [ ] Test that FAILS on pre-change behavior (link the red run or paste output)
- [ ] Command run + verbatim output for the primary claim
- [ ] Screenshot for any UI change
## Scope
- [ ] One-sentence purpose (above) covers every file in the diff

The same principle scales down into the agent session itself: Anthropic's docs describe a verification ladder — an in-prompt check, then a /goal condition, then a deterministic Stop-hook gate, then a verification subagent that tries to refute the result. Steps 6 and 7 use the top two rungs; the point here is that "it works" is never accepted as a sentence, only as output.

4. Run the machine pass first, human pass second

Reviewer attention is the scarce resource, so spend the cheap pass first. We run Claude Code's /code-review on every branch before a human looks: it reviews the branch's commits ahead of upstream plus uncommitted changes, finds correctness bugs and reuse/simplification/efficiency issues, and can target a file, PR, branch, or ref range. Effort levels are a real dial — low/medium return fewer, high-confidence findings; high and max go broader. --comment posts findings as inline PR comments; --fix applies them to the working tree. (It was renamed from /simplify at v2.1.147, if you're looking for it under the old name.)

# machine pass, medium effort, findings as PR comments
claude /code-review 123 --comment
# heavier variants when the diff warrants it:
claude /code-review ultra          # fleet of reviewers in a remote sandbox;
                                   # every finding independently reproduced, 5–10 min
claude ultrareview --json          # CI mode: exits 0/1 on findings

Two heavier tiers exist when a diff deserves them. Ultrareview (research preview since v2.1.86) runs a fleet of reviewer agents in a remote sandbox where "every reported finding is independently reproduced and verified" — 5–10 minutes, three free runs on Pro/Max then roughly $5–20 per review, with a claude ultrareview CI mode that exits 0/1 and emits --json; note it's not available on Bedrock/Vertex or under ZDR. And Anthropic's managed Code Review GitHub service (launched 2026-03-09, Team/Enterprise research preview, ~$15–25/review) runs multi-agent parallel analysis with a verification step that filters false positives, labels findings Important / Nit / Pre-existing, triggers on @claude review, and — deliberately — never blocks merge: it posts a neutral check and exposes bughunter-severity JSON so you decide the gate (step 7). Its REVIEW.md file is injected at highest priority and is where you redefine severities, cap nits, and set the verification bar — e.g. "behavior claims need a file:line citation." CLAUDE.md violations surface as nits.

Anthropic's internal numbers on this pipeline (self-reported, so attributed as such): before, 16% of PRs got substantive review comments; with the machine pass, 54% — and under 1% of findings are marked incorrect. If your stack is elsewhere, the same slot is filled by GitHub Copilot code review (agentic, severity-labeled, analysis-depth tiers), CodeRabbit (the most-installed review app, diff-level annotations), Greptile (full-codebase context, free tier since 2026-06), or Cursor Bugbot ($40/user/mo with Autofix cloud agents) — the workflow doesn't change, only the command. The human pass then reads what survived the machine pass, which is step 5.

The AI review pass above catches correctness and style; it does not catch what the code executes as. Extend the same machine pass with three deterministic gates that need no model at all — after typecheck, lint, and unit tests, still before any human reads the diff. In CI, chain all three onto the existing pass (&&) so the first failure stops the PR. This is layer 4 of the AI Quality Stack, installed here in full.

# SAST - static analysis for the vulnerability classes agents reintroduce
semgrep scan --config auto
# IaC scan - the agent's Terraform diff gets the same scrutiny as its app code
checkov -d infra/
# dependency audit - catches a vulnerable package pulled in to satisfy the task
npm audit --audit-level=high   # or: pnpm audit --audit-level high

Semgrep (free CLI, no login or account required) is the free-tier answer to the security-flaw rate cited above: pattern-based static analysis for the injection flaws, hardcoded secrets, and other OWASP-class defects that land in AI-generated diffs at roughly the same rate regardless of how clean the code reads. It is scope-limited on purpose — supply-chain (SCA) and secrets scanning sit behind Semgrep's paid AppSec Platform; the free CLI is SAST only, which is exactly the gap this step fills. Checkov (maintained by Prisma Cloud, Palo Alto Networks) reads the diff's Terraform and catches what an agent gets wrong in infrastructure the same way it gets application code wrong: a public S3 bucket, a security group open to the world, an IAM policy with a wildcard resource — findings a reviewer skimming HCL will not reliably catch by eye, caught here before a human ever opens the file. npm audit (or pnpm audit --audit-level high — space-separated, not =) targets the failure mode specific to agents solving a task by reaching for a new dependency: a vulnerable transitive package added in the same diff that shipped the feature, flagged before it reaches whatever dependency review your team still does by hand.

5. Trace one critical path by hand

The human pass is not a line-by-line read — the machine already did that. It is a single end-to-end trace of the diff's most critical path, because the machine passes are weakest exactly where AI-written code fails most convincingly: hallucinated correctness. Code that reads right, compiles, and is wrong. GitHub's reviewer checklist for AI-generated code names the concrete checks: do the imports resolve, do the called methods actually exist, do the mocks test behavior or merely test the wiring. Pick the one path where a bug costs the most — the auth check, the money movement, the data deletion — and walk it: entry point to side effect, boundary conditions at each hop, permission checks where trust changes hands.

While tracing, watch for reuse blindness: agents reimplement utilities that already exist three files away — the per-diff face of GitClear's duplication curve. A reimplemented helper isn't a style nit; it's a second copy that will drift from the first. The fix is a comment pointing at the existing util, and (step 8) a rule so the class of finding dies.

# the trace, made mechanical — for each hop on the critical path:
# 1. does this import/function exist?   git grep -n "functionName"
# 2. boundary: what happens at 0, empty, max, unauthorized?
# 3. is there already a util for this?  git grep -n "similarVerb" src/lib/

6. Split writer and reviewer contexts

The agent that wrote the code cannot grade it — not as an ethics rule but as a mechanics one. Anthropic's best-practices docs state the reason: "a fresh context improves code review since Claude won't be biased toward code it just wrote." The same context window that produced the diff contains the reasoning that justified it; asked to review, the model re-reads its own rationale and agrees with itself. So the reviewer is always a fresh context — a subagent or a new session — and it is framed adversarially: the docs describe having "a fresh model try to refute the result, so the agent doing the work isn't the one grading it."

Two framing rules make the fresh reviewer useful instead of noisy. First, the refute framing: the task is "find where this claim is false," not "review this code." Second, the over-reporting guard, straight from the docs: "tell the reviewer to flag only gaps that affect correctness" — an unconstrained reviewer pads its report with style findings to look thorough, and the real finding drowns.

# .claude/agents/adversarial-reviewer.md
---
name: adversarial-reviewer
description: Fresh-context reviewer. Tries to refute the diff's claim.
---
You did not write this code. The author claims: {CLAIM}.
Your job is to refute it. Trace the critical path; check boundaries,
permissions, and that called symbols exist. Flag ONLY gaps that affect
correctness — no style, no preferences. Every finding needs file:line.

This is first-person practice, not theory: this repo's merges are gated by exactly this pattern — fresh-context reviewer subagents plus adversarial verification — and the findings that survive the "refute it" framing are reliably the ones worth a human's time.

7. Gate deterministically — advisory rules don't gate

Everything so far produces findings; something has to refuse the merge. Anthropic's docs draw the distinction cleanly: hooks are deterministic, CLAUDE.md is advisory. A rule the model is asked to follow is a suggestion; a gate that runs as code is a gate. Inside the agent session, that's a Stop hook — a deterministic check (tests, lint, typecheck) that blocks the agent from declaring itself done until the check passes (it can be overridden only after 8 consecutive blocks, which is the escape valve, not the norm). In CI, it's an exit code: claude ultrareview --json exits 0/1, and the managed Code Review service's bughunter-severity JSON exists precisely so you can fail the build on Important findings even though the service itself never blocks merge.

# .claude/settings.json — the agent can't stop until the suite is green
{ "hooks": { "Stop": [ { "hooks": [ { "type": "command",
      "command": "npm run typecheck && npm test" } ] } ] } }

# CI — the org decides the gate, not the review service
claude ultrareview --json | jq -e '[.findings[] | select(.severity=="important")] | length == 0'

The principle generalizes past any one tool: every rule you actually rely on must exist as something that can return a nonzero exit code. The trust-then-verify gap is a named failure mode in Anthropic's own guidance — "if you can't verify it, don't ship it" — and a gate that lives only in prose is a gap wearing a rule's clothes.

8. Feed findings back so the class of bug dies

A finding fixed once is a cost; a finding converted into a rule is an asset. When the same class of issue shows up in a second review, it stops being a review comment and becomes: a CLAUDE.md rule (advisory — shapes what the agent writes next time; violations surface as nits in the managed service), a REVIEW.md rule (shapes what the reviewer hunts for and the evidence bar it enforces), or — best of all — a lint rule or test, which is deterministic and free forever.

# REVIEW.md — injected highest-priority into the managed review
## Severity overrides
- Any change under .github/workflows/ that removes a step: Important, always.
- Reimplementing a helper that exists in src/lib/: Important, cite the existing file.
## Verification bar
- Behavior claims need a file:line citation.
- Max 3 nits per review; prefer zero.

Route by determinism: if a machine can check it, make it lint or a test; if only a model can check it, put it in REVIEW.md; if it's about how code should be written in the first place, CLAUDE.md. The measure of a review process is not findings per PR — it's whether last quarter's most common finding is extinct.

As built: reviewing our own injection gate

On 2026-07-24/25 this repo put an LLM-based prompt-injection gate for the coach through several review rounds — machine passes, fresh-context adversarial subagents, and reviewers from more than one vendor. Three things came out of it that change how the steps above get run.

A severity rating is a claim to verify, not a fact to accept

A reviewing agent flagged a JSON-parsing weakness in how the classifier's output was handled, rated it "Minor, defer", and explained the trigger as trailing prose containing a closing brace. That rating was accepted without checking. Executing the function took about thirty seconds and showed the real trigger was far wider: any opening brace appearing before the verdict object. Four cases were run. A clean verdict parsed. Prose with no braces parsed — which is exactly why the existing unit test passed. A preamble quoting learner text that contained a brace failed open. And the model's own reasoning-channel text landing in the content failed open. Fail-open on a security gate means the turn proceeds unscreened, and two of those four triggers are partly influenceable by the person being screened.

# before accepting a severity, run the parser on the cases the finding implies:
# 1. clean verdict only                     -> the happy path
# 2. prose, no braces, then the verdict     -> what the existing test covered
# 3. quoted user text containing a brace    -> untested; failed open
# 4. reasoning-channel text in the content  -> untested; failed open
# 3 or 4 failing = the rating was wrong, not the finding

Reviewers produce two things — a finding and a severity — and the severity is the softer of the two. It is generated from a reading of the code, not from running it. Step 3's evidence contract applies to the reviewer as much as to the author: on anything touching a security or money path, reproduce the trigger the finding describes before you agree to defer. Thirty seconds of execution beats a confident rating.

Diff-scoped review has a structural blind spot

Three of the highest-value findings in this review were invisible to every diff-based reviewer, because the defect was not in any changed line:

• A committed document referenced a spec file that was never tracked by git — the reference resolved for the author and for nobody else.
• A public privacy page became inaccurate because a new code path introduced a data processor the page did not list. The page itself was unchanged, which is precisely why no diff review looked at it.
• An npm script that aggregates the project's gates was never updated to include the new gate, so the new checks did not run in the standard invocation.

A diff shows what changed, not what the change invalidated. Anything whose correctness depends on the changed code — untracked or generated files, legal and compliance pages, aggregate scripts, docs asserting current behavior — needs a pass scoped to consequences rather than to lines. The sibling case on this platform is already known: a text-wide brand rename updated 618 instances across 110 files, but the social preview images are PNGs, so every shared link served the old wordmark for eleven days while every automated gate stayed green. Same failure class — a completeness check that only covers the representation it can parse.

Different models find disjoint sets, so run more than one

Across the rounds the highest-severity findings barely overlapped. An adversarial pass by one model family found the parse fail-open; a different vendor's reviewer found the undisclosed data processor; a scope-oriented pass found the untracked spec file. Almost no overlap at the top of the list. So a second reviewer is not redundancy, and it should not be judged on agreement rate — low overlap is the return, not evidence that one reviewer is weak. The inverse holds too: consensus across models is not proof of correctness, because a wrong-but-plausible finding reproduces across every reviewer handed the same misleading context.

As built: reconciling two reviewers

Running a second reviewer is the easy half. The half that gets skipped is deciding what two partly-disagreeing reports mean, and by the time you are reading them both you are tired and inclined to take whichever one is shorter. Two earlier rounds in this repo — the 2026-07-12 positioning review and the 2026-07-13 pattern-library release — settled on one artifact for that, and it is cheap enough to be worth copying.

The consensus table, and the rule that keeps it from becoming a meeting

Fix the review dimensions before either reviewer runs. Give each reviewer its own column. Add a last column that reads CONFIRMED where they agree and DISAGREE where they do not, with the substance of the split written out rather than averaged into a middle position. The engineering round of that review produced this:

  Dimension                       Claude   Codex    Consensus
  1. Architecture sound?          YES      YES      CONFIRMED
  2. Test coverage sufficient?    NO       NO       CONFIRMED FAIL
  3. Performance risks addressed? YES      YES      CONFIRMED
  4. Security threats covered?    YES      YES      CONFIRMED
  5. Error paths handled?         MOSTLY   NO       DISAGREE
  6. Deployment risk manageable?  YES      YES      CONFIRMED

The table earns its keep only with a routing rule attached, or every DISAGREE becomes an escalation. Ours: a disagreement about a premise — this is the wrong thing to build, this price has no evidentiary basis — goes to whoever owns that decision. Everything else the dispatcher resolves itself and records in the same document with the reason it turned on, so the resolution is auditable months later instead of living in a transcript. On the strategy round, six dimensions came back five CONFIRMED and one DISAGREE; that split and two premise challenges went up, and the remaining resolutions were auto-decided into a numbered decision table. Row 5 above is the shape you want to see: not two reviewers agreeing, but a disagreement forced into a line item that has to be closed before deploy.

What the second vendor actually caught

The concrete save from the pattern-library round: one of our own pages taught Claude Code's permission model as deny and ask rules apply in every mode, including bypass, so a forbid is genuinely unbypassable, and described the Agent SDK's canUseTool callback as where your boundary decision runs. A reviewer running on a different vendor's model rejected both readings. The corrected text is the one now on the page: explicit deny rules are the hard forbid boundary while the mode sets only the default posture, and canUseTool is reached only for calls the rules have not already resolved — an auto-approved call never reaches it, which makes it an ask-tier classifier rather than a forbid boundary. Mandatory enforcement stays in deny rules, or in a PreToolUse hook that runs on every call.

That text was internally consistent, well cited, and wrong about a security boundary — precisely the finding a single reviewer is worst at, because nothing in the passage contradicts anything else in the passage, and a model that read it once reads it the same way again. Fixed in commit 84a6991, before deploy.

Five days later the same gap showed up in the other direction, on this page. Two example invocations, npx semgrep and npx ajv, were written into a plan from memory before the research step that would have checked them; the writers transcribed them faithfully, and the fake commands cleared five review passes and a merged PR. A cross-model pass caught them afterwards, along with example gates whose exit codes were inverted — commands that report success on the failing case. Two lessons, and the second one is the reason this subsection exists: runnable commands on a page are code and need a cited source per command (step 3's evidence contract, applied to your own examples), and five passes from one model family are correlated samples rather than coverage. If a claim would be expensive to have wrong, the second opinion has to come from a different model, not a fresh session of the same one.

Artifacts this workflow produces

• REVIEW.md — severity redefinitions, nit caps, and the verification bar, injected highest-priority into machine review.
• The reviewer-subagent definition — fresh-context, adversarial refute framing, correctness-only scope, file:line citations required.
• The Stop-hook gate — deterministic typecheck+test gate the writing agent cannot talk its way past.
• The PR template — evidence contract: failing-test-first, command output, one-sentence scope.
• The severity-JSON CI gate — the pipeline step that fails the build on Important findings, since the review services deliberately never block merge themselves.
• The consensus table — dimensions fixed before the reviewers run, one column per reviewer, CONFIRMED/DISAGREE in the last, and a routing rule that sends only premise-level disagreements upward.

Pitfalls

• Reviewing the source before the gates. A weakened workflow file makes every subsequent green check a lie. CI, test, and coverage diffs first, always.
• Letting the writer grade itself. Same context = same bias; the model re-reads its own rationale and approves it. Fresh context or it isn't a review.
• Accepting assertions as evidence. "Tests pass" is a sentence. The test output, the command and its return, or the screenshot — or it didn't happen.
• Unconstrained reviewers. A reviewer not told "correctness gaps only" pads with style findings, and the one real bug drowns in nits.
• Advisory rules doing a gate's job. CLAUDE.md shapes behavior; it does not refuse merges. Anything you rely on must exit nonzero somewhere.
• Rubber-stamping small-looking diffs from big prompts. Diff size is a proxy, not an absolution — a 30-line auth change still gets the hand trace (step 5).
• Treating reuse blindness as style. The duplicate helper is tomorrow's divergent behavior. It's a correctness finding with a delay on it.
• Fixing findings without killing the class. If the same finding appears twice and there's no new rule or lint, the review process is a treadmill, not a ratchet.
• Accepting a reviewer's severity as a fact. The finding is evidence; the severity is a guess about blast radius, read off the code rather than run. On a security or money path, reproduce the trigger before you agree to defer.
• Scoping the review to the diff. The diff shows what changed, not what the change invalidated — untracked files, compliance pages, aggregate gate scripts, docs asserting current behavior. Review consequences, not lines.
• Judging a second reviewer on how much it agrees. Disjoint findings across models are the return, not a defect — and agreement is not proof, since a wrong-but-plausible finding reproduces everywhere the same misleading context does.
• Collecting two reports and never reconciling them. Two reviewers create a decision you now have to make. Without fixed dimensions, a CONFIRMED/DISAGREE column, and a rule for which splits escalate, the tie gets broken by whichever report is shorter.
• Buying more passes instead of a different model. Repeat passes from one model family are correlated samples. Five of them here missed two invented CLI invocations and a set of inverted exit codes in example gate commands.

When not to use this workflow

Don't run the full eight steps on throwaway code: spikes, one-off analysis scripts, prototypes that will be deleted rather than merged. The discipline exists to protect a trunk that ships; code that never merges needs only step 5's instinct — don't act on its output without a sanity check.

And don't use review as a substitute for specification. If every PR from an agent needs heavy correction, the defect is upstream — in the task framing, the CLAUDE.md, or the missing eval harness. That's the territory of eval-first development and the Claude Code workflow: get the writing loop right and this workflow gets cheaper, because review is the backstop, not the process.

Changelog

• 2026-08-10 — added the reconciliation as-built section: the consensus table and its premise-only escalation rule, the cross-vendor canUseTool catch (fixed in 84a6991), and the five-passes-one-family miss on this page's own example commands; two matching pitfalls and the table added to the artifacts list.
• 2026-07-25 — added the as-built section from this repo's coach injection-gate review (severity is a claim to verify, consequence-scoped review beyond the diff, disjoint findings across models) plus three matching pitfalls.
• 2026-07-18 — step 4 (machine pass) extended with three deterministic gates: Semgrep SAST, Checkov IaC scan, npm/pnpm dependency audit; cross-linked as layer 4 of the AI Quality Stack.
• 2026-07-13 — initial version, verified against Claude Code /code-review docs and 2025–26 industry studies.