Don’t Let Claude Grade Its Personal Homework

I had a handful of parallel Claude brokers write me up a technique folder for a facet challenge. The pull request appeared nice: ten paperwork, all neatly cross-linked, and it learn effectively. Then the automated reviewer posted its findings, and Claude, to its credit score, didn’t object:

The complete audit got here again: 30 of the 95 relative hyperlinks pointed at recordsdata that didn’t exist. ./market.md as an alternative of market-analysis.md, ./monetization.md as an alternative of business-model.md, some recordsdata had been even made up totally! Skimming that PR as a human, I might have permitted it, since each one in every of them appeared believable. Fortunately, my automated reviewer caught it as a result of it resolved each hyperlink as an alternative of judging whether or not the names “vibed” proper.

To be honest, Claude doesn’t have any dangerous intent behind this. A language mannequin merely generates essentially the most believable continuation, and typically essentially the most believable continuation is fiction delivered with full confidence. From the studying facet, although, that distinction brings little consolation. Assured and improper reads precisely like assured and proper.

This put up is concerning the system I’ve in place to assist me overview Claude’s errors and hallucinations: a Codex agent working in GitHub Actions that critiques each pull request. It walks you thru my setup, why cross-provider overview beats each self-review and a human-reads-everything overview, methods to implement reminiscence and accountability within the overview loop, the associated fee and reliability learnings I gathered alongside the best way, and the precise code I exploit to make this work.

Agent throughput broke the human overview loop

Hallucinations by themselves are a manageable drawback, since a really cautious overview or practical evaluation would catch them. Nevertheless, what broke over the previous yr is the amount that wants reviewing. A single engineer now runs a number of agent periods in parallel, and every of them can produce a thousand-line diff within the time it takes to refill your espresso. Producing code has change into the quick and simple half; reviewing it truthfully is what eats the day. That makes the human reviewer the bottleneck of the entire supply pipeline, and the bottleneck is at all times the subsequent factor it’s best to automate.

That bottleneck invitations two acquainted failure modes. Both you approve regardless of the brokers produce with out actually studying it, and the damaged hyperlinks, phantom parameters, and subtly improper edge instances ship to manufacturing. Otherwise you nonetheless insist on studying each line your self, and your quick and sensible AI brokers work even lower than you do. The primary is negligent, the second wasteful, and most groups oscillate between each relying on the deadline.

The conclusion I maintain touchdown on: our overview course of has to scale with the pace we’re producing code. The human ingredient stays, however the human reviewer’s focus ought to go to the elements that actually matter. The grind of checking whether or not hyperlinks resolve, whether or not the brand new parameter exists, whether or not there’s a bug or vulnerability, and whether or not the code adjustments match what was initially requested, all of that has to run mechanically on the identical cadence because the coding brokers, even earlier than an individual ever spends a second reviewing the code.

Perception: hallucinations are a continuing, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying pace, so the very first overview move needs to be automated.

Choose a reviewer that doesn’t share the writer’s blind spots

The plain first concept to stop these errors is to have the agent overview its personal work… however it is usually the weakest one. A hallucination that made it into the diff is, by definition, believable to the mannequin that produced it. Asking the identical weights to seek out errors within the code, particularly inside the identical already-biased coding session, normally reproduces the precise blind spots that allow the errors by means of.

This instinct has analysis behind it. Panickssery et al. confirmed at NeurIPS 2024 that LLM evaluators acknowledge and favor their very own generations: the higher a mannequin is at figuring out its personal output, the extra generously it scores it. Comply with-up work ties this self-preference bias to familiarity: AI judges rating textual content larger when it reads as predictable to them, and nothing reads as extra predictable to a mannequin than the textual content it simply wrote itself.

Totally different LLM suppliers prepare on completely different information, with completely different recipes and completely different suggestions loops. No supplier manages to get rid of errors, however the type of errors a mannequin makes traces again to the way it was educated, and that differs per supplier. That decorrelation is strictly what you need in your overview agent. Within the yr that I’ve used this setup, OpenAI’s Codex has persistently flagged points in Claude-written PRs that Claude’s personal overview handed with out blinking, from invented filenames to total code blocks that had nothing to do with the preliminary request, or that launched outright safety violations.

One conclusion you can’t take from that is that Codex is the higher engineer and it’s best to hand it the keyboard as an alternative of Claude. Codex misfires pretty repeatedly, too! Simply in a unique course than Claude does. The ability sits in working multimodel: combining suppliers inside the identical workflow, the place the fashions fill one another’s gaps.

One other incorrect conclusion is that Codex’s critiques are good and could be accepted blindly. Hallucinations occur in critiques too, so deal with the overview as advisory. Difficult the writer (be it Claude or your self) places the concentrate on potential blind spots, which is the proper sign to zoom in and replicate. It stays the writer’s job to personal the ultimate high quality, and typically meaning writing again to Codex that its overview was improper. Extra on this mechanism later within the put up.

Perception: an writer’s bugs are by definition believable to the writer, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in a different way, and completely different is what catches what you miss.

The pipeline: one LLM name wrapped in atypical engineering

Conceptually, an LLM reviewer could be written in a single line: “right here is the code diff, please overview it”. Each OpenAI and Anthropic ship hosted shortcuts for precisely that: OpenAI has a GitHub overview integration that auto-reviews PRs, and Anthropic has claude-code-action. They’re wonderful beginning factors, however I want to run my very own motion as an alternative, as a result of I wish to personal the immediate, the overview lifecycle, the merge gating, and the mannequin pin. All these features matter when your day-to-day turns into managing and reviewing brokers.

So what am I utilizing then, precisely? Two recordsdata that reside within the repository they overview:

.github/
├── workflows/
│   └── pr.yml            # orchestration: triggers, context, retries, posting
└── actions/
    └── codex-pr-review/
        └── motion.yml    # the reviewer: pinned Codex name + overview immediate

The runtime circulate is brief. Each time a PR opens or receives a push, pr.yml checks out the code, collects the PR’s full dialog historical past, and fingers each to the composite motion. The motion runs Codex over the diff and returns its findings as a single message. The workflow then posts that message as a single, repeatedly up to date remark within the PR dialog, and publishes a commit standing that stays crimson whereas findings stay unresolved: the go/no-go sign sitting subsequent to the remainder of CI.

On the middle of all of it sits one deceptively easy step, the one line that really touches an LLM:

- makes use of: openai/codex-action@v1
  with:
    openai-api-key: ${{ inputs.openai-api-key }}
    immediate: |
      # the overview contract: what to overview, methods to observe findings,
      # and methods to report again (unpacked within the subsequent part)

All the things else exists to run that single step reliably, and to verify its suggestions really lands on the PR. Right here is the whole workflow:

identify: PR Reviewer

on:
  pull_request:
    branches: [dev] # all work rides function branches: function -> dev -> major
    varieties: [opened, synchronize, reopened]

# a brand new push cancels the in-flight (costly) overview of the earlier commit
concurrency:
  group: pr-reviewer-${{ github.occasion.pull_request.quantity }}
  cancel-in-progress: true

jobs:
  codex:
    if: github.occasion.pull_request.consumer.login != 'dependabot[bot]'
    runs-on: ubuntu-latest
    timeout-minutes: 15 # regular overview takes a couple of minutes, so 15m is greater than sufficient
    permissions: # the token can learn code and put up suggestions, however can by no means push or merge
      contents: learn
      points: write
      pull-requests: write
      statuses: write
    steps:
      # the /head ref updates synchronously on each push;
      # the auto-generated merge ref can race the checkout
      - makes use of: actions/checkout@v7
        with:
          ref: refs/pull/${{ github.occasion.pull_request.quantity }}/head

      # be certain that each diff endpoints exist domestically, so Codex can resolve base...head
      - identify: Pre-fetch base and head refs for the PR
        run: |
          git fetch --no-tags origin 
            ${{ github.occasion.pull_request.base.ref }} 
            +refs/pull/${{ github.occasion.pull_request.quantity }}/head

      # suggestions lives on three surfaces: concern feedback, critiques, and inline feedback
      - identify: Fetch PR dialog
        id: dialog
        makes use of: actions/github-script@v9
        with:
          result-encoding: string
          script: |
            const prNumber = context.payload.pull_request.quantity;
            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };
            const [comments, reviews, inline] = await Promise.all([
              github.paginate(github.rest.issues.listComments,
                { ...opts, issue_number: prNumber }),
              github.paginate(github.rest.pulls.listReviews,
                { ...opts, pull_number: prNumber }),
              github.paginate(github.rest.pulls.listReviewComments,
                { ...opts, pull_number: prNumber }),
            ]);
            const all = [
              ...comments.map(c => ({
                date: c.created_at,
                text: `@${c.user.login}: ${c.body}`,
              })),
              ...reviews.filter(r => r.body).map(r => ({
                date: r.submitted_at,
                text: `@${r.user.login} (review ${r.state}): ${r.body}`,
              })),
              ...inline.map(c => ({
                date: c.created_at,
                text: `@${c.user.login} (on ${c.path}): ${c.body}`,
              })),
            ];
            all.kind((a, b) => new Date(a.date) - new Date(b.date));
            return all.map(e => e.textual content).be part of('n---n');

      # secrets and techniques don't journey with the yml, talk loudly on lacking keys
      - identify: Fail quick on lacking OPENAI_API_KEY
        env:
          OPENAI_API_KEY: ${{ secrets and techniques.OPENAI_API_KEY }}
        run: |
          if [ -z "${OPENAI_API_KEY}" ]; then
            echo "::error::OPENAI_API_KEY secret is just not set for this repository."
            exit 1
          fi

      # retry wrappers can't wrap `makes use of:` steps, so retry manually: transient
      # failures (a 401 from a recent runner token, a hiccup reaching OpenAI)
      # shouldn't flip the entire run crimson
      - identify: Run Codex (try 1)
        id: codex_try1
        makes use of: ./.github/actions/codex-pr-review
        continue-on-error: true
        with:
          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}
          dialog: ${{ steps.dialog.outputs.outcome }}

      # give transient failures a second to clear earlier than retrying
      - identify: Wait earlier than Codex retry
        if: steps.codex_try1.end result == 'failure'
        run: sleep 20

      - identify: Run Codex (try 2)
        id: codex_try2
        if: steps.codex_try1.end result == 'failure'
        makes use of: ./.github/actions/codex-pr-review
        with:
          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}
          dialog: ${{ steps.dialog.outputs.outcome }}

      - identify: Put up the overview and publish a standing
        makes use of: actions/github-script@v9
        env:
          REVIEW: >-
            ${}
        with:
          script: |
            const MARKER = '';
            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };
            const prNumber = context.payload.pull_request.quantity;
            const uncooked = course of.env.REVIEW;
            if (!uncooked) return;
            const physique = `${MARKER}n${uncooked}`;

            // one canonical remark, edited in place on each push
            const feedback = await github.paginate(
              github.relaxation.points.listComments, { ...opts, issue_number: prNumber });
            const current = feedback.discover(c =>
              c.consumer?.login === 'github-actions[bot]' && c.physique?.consists of(MARKER));
            const posted = current
              ? await github.relaxation.points.updateComment(
                  { ...opts, comment_id: current.id, physique })
              : await github.relaxation.points.createComment(
                  { ...opts, issue_number: prNumber, physique });

            // go/no-go: a commit standing that stays crimson whereas findings are open
            // (a lacking trailer reads as "overview posted", by no means as zero findings)
            const m = uncooked.match(/`, the place N counts
              the unresolved rows and M what number of of these are severity Excessive.
              At all times embody it, even when N is 0: CI parses it into the
              standing verify.

          Pull request title and physique:
          ----
          ${{ github.occasion.pull_request.title }}
          ${{ github.occasion.pull_request.physique }}

          Dialog historical past:
          ----
          ${{ inputs.dialog }}

Three design selections carry the burden right here.

• The overview is stateful. The Suggestions Abstract desk tracks each discovering ever raised on the PR, resolved or not, throughout pushes. Mixed with the edit-in-place remark from the workflow, a PR carries precisely one overview artifact that displays the complete lifecycle, as an alternative of 5 stale feedback that every replicate an replace within the overview course of. Anybody opening the PR, human or agent, sees the total historical past at a look.

• Declines are honored. As a result of Codex reads the entire dialog, the writer can reject a discovering in writing, and the rejection sticks throughout overview cycles. That is the mechanism that makes an imperfect reviewer workable: false positives price one written reply as an alternative of an countless nag loop, and the objection stays on report for the subsequent reader.

• The decision is machine-readable. The trailing line will get parsed by the workflow right into a Codex overview commit standing. The result’s two impartial indicators on each PR: the workflow verify tells you the overview ran, and the standing tells you whether or not something remains to be open. Crimson standing means unresolved findings; inexperienced means the whole lot is fastened or explicitly declined. I intentionally maintain the standing non-blocking, because it solely features to tell. The choice to merge or not stays open, and stays human.

Word how this contract asks for greater than typo looking. Codex critiques the diff towards the PR’s said intent, ranks each discovering by severity, and attaches a concrete proposed repair, the type of overview that repeatedly surfaces architecture-level points no linter would ever produce.

Perception: construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.

Shut the loop: the PR turns inexperienced earlier than you even look

The reviewer is just half the story once you take a look at the larger engineering image. The opposite half is instructing the writer agent to reply to it, so the overview round-trip doesn’t change into your job (keep in mind: at all times attempt to take away the bottleneck). In my repos I exploit three Claude Code expertise to help me, written as plain markdown instruction recordsdata underneath .claude/expertise/:

• /pr-open creates the PR: analyzes the department’s commits, runs validation domestically, writes an sincere title and physique, and targets the proper base. The PR physique issues greater than it appears, because the reviewer reads it because the assertion of intent it critiques towards. I’ve it spell out the total function request and reference the Linear ticket it implements, so each the reviewer and any human arriving later get the whole image with out leaving the PR.
• /pr-iterate handles one overview spherical: sweeps all three suggestions surfaces, then triages each merchandise right into a repair, a decline with reasoning, or a defer. Fixes are carried out and validated domestically; declines change into PR feedback explaining why. The reviewer is advisory, and “adequate to merge” is a legitimate verdict as soon as the PR meets its said scope.
• /pr-babysit is the watchdog round all of it: watch the CI, repair crimson checks, run an iterate spherical when new suggestions lands, and loop till each verify is inexperienced and each discovering is fastened or declined. Its terminal state is “merge-ready”, and it stops to report reasonably than ping-pong commits when the identical verify fails twice on the identical root trigger.

One ordering rule that you just and your brokers ought to at all times be mindful: reply first, push after. The overview kicks off on each push, and it really works with the context that exists at that actual second. Something not but written down (a decline, a repair abstract, a remark of your personal) doesn’t exist for that overview spherical, and the reviewer will fortunately re-raise each merchandise it can’t see a response to.

The top state is a tidy division of labor. Claude writes and defends, Codex challenges and tracks, the loop runs till the PR is inexperienced with a clear suggestions desk. My queue incorporates solely PRs which can be already merge-ready, every carrying a overview path I can audit in a single scroll: what was discovered, what was fastened, and what was declined and why.

Perception: automate either side of the overview. When the writer agent should repair or contest each discovering in writing earlier than it may possibly push, the human queue solely ever incorporates merge-ready PRs with an auditable overview path.

What a round-trip appears like in observe

To make the mechanism concrete, right here is one change from a backend PR of mine. The PR fastened a capability grid that wrote extra cells than a hardcoded batch cap allowed. The primary overview spherical got here again with two findings:

## Suggestions Abstract

| Situation                           | Severity | Abstract                                  | Resolved |
| ------------------------------- | -------- | ---------------------------------------- | -------- |
| Truncation past the batch cap | Excessive     | Cells previous the cap are silently dropped. | ❌       |
| Cap duplicates a platform restrict | Medium   | Transfer the cap into app config.            | ❌       |

The Codex overview standing turned crimson. Claude carried out the primary discovering for actual, then declined the second in a PR remark (actual remark, anonymized):

Addressed the Excessive discovering (hard-coded row cap within the information layer, inflicting avoidable 502s).

_fetch_capacity_grid now adapts as an alternative of failing: on a brief web page (which, as a result of the grid is dense at precisely days rows/particular person, unambiguously means the deployed cap truncated the batch) it halves batch_size and retries the identical slice, shrinking till pages match. A brief web page solely turns into a loud 502 as soon as batch_size == 1 [...]

On the secondary suggestion (transfer the cap to config): deliberately not carried out. The actual cap lives within the database’s personal hosted API settings, not an app env var. Mirroring it into app config would add a second supply of fact that would silently drift from the database. Adaptive batching removes the necessity to know the precise worth in any respect, which is the stronger repair. Present capacity-grid DB check nonetheless passes; lint and type-check inexperienced.

Reply first, push after, and have Codex re-review:

## Suggestions Abstract

| Situation                           | Severity | Abstract                          | Resolved |
| ------------------------------- | -------- | -------------------------------- | -------- |
| Truncation past the batch cap | Excessive     | Fastened by way of adaptive batching.     | ✅       |
| Cap duplicates a platform restrict | Medium   | Declined: duplicate would drift. | ✅       |

Each rows resolved, one by a repair and one by a written decline that the reviewer accepted and recorded. That second row is the complete mechanism in a single line: no churn was carried out simply to silence a remark, and the reasoning is now a part of the PR’s historical past for whoever reads it subsequent.

The learnings ledger: what it prices and the place it bites

This setup didn’t come collectively in a single go. Most of its particulars grew out of surprises and useless ends alongside the best way, so I’m writing the learnings down right here, in tough order of cash saved, to spare you from driving into the identical partitions:

1. Pin the mannequin. The mannequin enter of codex-action is non-obligatory, and leaving it empty means “no matter Codex presently defaults to”. I discovered this on my invoice when the default silently moved to a more moderen, pricier mannequin (gpt-5.5). Pins clear up this drawback, however want upkeep in return; presently I exploit gpt-5.3-codex, which is already marked deprecated on the Codex fashions web page on the time of writing (whereas remaining obtainable on the API). I revisit the pin intentionally as an alternative of letting the default determine for me.
2. Cap the trouble at medium. Increased reasoning effort catches marginally extra points for considerably extra time and money. Velocity issues greater than it appears: the overview sits contained in the agent’s iterate loop, so its latency multiplies throughout each spherical of each PR, and your personal ready time is a part of the value. Medium has been the issues-per-dollar and issues-per-minute candy spot. Sadly, no effort stage catches the whole lot; that’s what the merge-owning human is for.
3. Retry the fast failures. A recent runner token often 401s, and the primary connection to the API often hiccups. Two makes an attempt with a gated fallback maintain these from turning the workflow crimson “for nothing”, which issues as soon as brokers (and your personal belief) deal with crimson as an actual sign.
4. Cancel outdated critiques. The concurrency block means solely the newest commit of a PR will get reviewed. With out it, an agent pushing three fixes in a row buys you three critiques, two of them for code that now not exists.
5. Evaluation towards the bottom that can obtain the merge. My PRs goal dev, so the overview diffs towards dev, and I maintain the function department synced with that base. A stale department will get diffed towards a base that has moved on, and the overview then stories phantom findings for adjustments the PR by no means made.
6. Characteristic branches are the prerequisite. All of this hangs off pull_request occasions. Direct pushes to dev or major bypass the reviewer, the standing, and the audit path totally. The one-branch-per-change circulate is what makes each change reviewable within the first place.
7. Assume PR textual content is untrusted enter. All the things the reviewer reads (the PR physique, the dialog, the code itself) is a prompt-injection floor, and the agent executes in your runner together with your API key. The least-privilege token caps the blast radius on the GitHub facet, and codex-action ships sandboxing and safety-strategy knobs for the runner facet. Additionally know that GitHub doesn’t expose secrets and techniques to workflows triggered from forked PRs, so this sample assumes same-repo branches from trusted contributors. Evaluation the belief mannequin earlier than copying it into an open-source repo.

You’re the editor-in-chief now

The purpose of this pipeline was by no means to take away the human from the loop, its intent is to uplevel you. I finished being the primary reader of each diff and have become the one that reads critiques, adjudicates the occasional standoff between two fashions, and owns the requirements written into the overview immediate. When Claude and Codex agree, the PR might be wonderful. Once they disagree, that disagreement is a precision-guided pointer at precisely the code that deserves my consideration. The place Claude typically acts sloppy, Codex tends to over-engineer, holding that stability in your personal fingers is a superb place to be.

Reviewing the critiques (so meta) is a greater job than proofreading agent output at agent pace, and it degrades gracefully. On a lazy day I merge inexperienced PRs on the energy of the path, on a cautious day I learn the suggestions desk and spot-check the declines. Both manner, nothing reaches the merge button on one mannequin’s assured say-so.

Key insights from this put up

1. Hallucinations are a continuing, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying pace, so the very first overview move needs to be automated.
2. An writer’s bugs are by definition believable to the writer, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in a different way, and completely different is what catches what you miss.
3. The mannequin name is the straightforward half. Reviewer reliability comes from atypical CI engineering round it.
4. Construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.
5. Automate either side of the overview. When the writer agent should repair or contest each discovering in writing earlier than it may possibly push, the human queue solely ever incorporates merge-ready PRs with an auditable overview path.

Remaining perception: work multimodel. Your brokers will typically hand you assured fiction, and fashions from completely different suppliers fill one another’s gaps, so that you don’t should. A second-provider reviewer with reminiscence and a contract, sitting between your brokers and your major department is a superb place to begin, however it isn’t the one one. Each time one mannequin’s output is about to matter, a second opinion from a unique lab is the most affordable insurance coverage you should purchase.

Comply with me on LinkedIn for bite-sized AI insights, In direction of Knowledge Science for early entry to new posts, or Medium for the total archive.