Who teaches the reviewer?
Code review was never mostly about catching bugs. When agents do the reviewing, the knowledge it used to pass between people has to be built in on purpose. What I built to try that, what about 70 agent reviews showed, and what a second model caught that 30 reviews missed.
You know the kind of review comment I mean. Somebody senior leaves it on a junior developer’s pull request, and it looks like this:
Not the ORM here. Use COPY. The import job took 40 minutes last September and timed out twice.
Two short sentences. They fix the change. They also hand over a piece of the team’s history that isn’t written down anywhere else, and the junior will remember that import job for years without ever having seen it.
Now make the reviewer an agent.
It might catch the slow insert (it knows bulk inserts through an ORM are slow). It can’t know about last September. And when the session ends, nothing it learned from your reply goes anywhere.
In my last article I wrote that review is where the work waits now: teams with heavy AI use merge far more pull requests, and the time they spend in review has grown fivefold.
The obvious move is to let agents review too, and teams are making it. I wanted to know what that move quietly drops, and whether I could build something to put it back. So I built it, and then I measured it.
Review was never mostly about bugs
In 2013, Alberto Bacchelli and Christian Bird studied code review at Microsoft. Developers told them finding defects was their main reason to review. Then the researchers sorted 570 real review comments by what they were actually about.
Defects came fourth. Fourth of nine categories, with 78 comments (14%), and most of those were small logic slips.
What kept coming up instead was learning. Knowledge transfer wasn’t even in their interview questions, and it came up anyway, unprompted, in every interview. Managers described review as “education for junior developers”. Google went further and built the teaching in on purpose: its readability process is an apprenticeship that runs through code reviews.
So review always did two jobs. It caught some bugs. And it moved knowledge between people: what this codebase is for, how we do things here, and why.
Agents are getting good at the first job. Meta now lets an automated system land low-risk changes with no human review at all.
In the paper describing it, that system reviewed 535,290 diffs and landed 331,720, with a third of the revert rate and a fiftieth of the incident rate of other diffs. (Those diffs were picked for being low risk, so it isn’t a fair comparison.)
Here’s the part this article is about. The authors name the cost themselves: “the trade-off between review efficiency and the knowledge diffusion that review provides.” They judge it favourable for now, and expect that “this balance may shift as automation coverage expands.”
The second job doesn’t move over to the agent on its own. Somebody has to build it in.
The idea: teach the reviewer, and let it teach back
Half of this is obvious. Let the human in the loop teach the agent. Every time you override a reviewer, reject a finding, or say “this looks wrong, but it’s on purpose”, that’s a lesson with a reason attached. Write it down where the next review will read it.
Several review products already do a version of this. Copilot, CodeRabbit, Greptile and Cursor’s Bugbot all turn human feedback into rules for their reviewer. The research on them points the same way: most of the measured gain comes from learning when to stay quiet, rather than from new knowledge.
The other half is the one I couldn’t find anyone doing: the way back.
If the senior’s correction goes into the agent’s memory instead of the junior’s head, the junior loses the lesson. So the memory has to be written for people too. Plain sentences, each with its reason and a pointer to the code. Then the same file that keeps the reviewer honest is also the junior’s reading list.
I made that a file in the repository, REVIEW-LESSONS.md. Each lesson looks like this:
## L-003 The finance report rounds half-even on purpose
- scope: shop/report.py
- kind: decision
- why: finance reconciles the report against the bank's statements, which round
half-even. Half-up made the totals drift by a few cents a month in 2025.
- evidence: shop/report.py:4-5 @ 3f2c1a9
- taught by: maya, review of PR #27, 2026-04-11
- status: activeNow, a memory that a reviewer obeys is also something an attacker can aim at, and a stale lesson is as dangerous as a wrong one. So the rules matter more than the format:
- Lessons are read from the protected base branch, never from the change being reviewed. Otherwise a pull request can rewrite the rules of its own review. One recent benchmark found that a fake “this was already approved” note was among the most effective ways to get a malicious change past weaker review agents.
- The human writes the why, in their own words. The agent can draft the scope and the evidence. A lesson without a reason is an opinion, and agents apply opinions badly.
- Every lesson points at code, and goes stale when that code changes. A stale lesson becomes a question for a person, not a rule.
- Anything that loosens a check needs a person and an expiry date.
- 1
A person makes a call, with a reason: “this looks wrong, but it’s on purpose.”
- 2
The reason becomes a lesson in REVIEW-LESSONS.md on the protected base branch, pointing at the code.
- 3
The reviewer reads it on the next change, and names who to ask.reads it
- 4
A junior reads the same file to learn why the code is the way it is.reads it
- 5
When the code it points at changes, the lesson goes stale and turns into a question for a person.
The loop this article argues for. Steps 3 and 4 read the same file: the reviewer to decide, the junior to learn.
What I built
Three Agent Skills that share that lessons file, one for each place I think a person still belongs in the loop:
- review-router decides how much human attention a change needs, judged from the change rather than its author, and says which three places to read first.
- spec-review checks a change against what was asked for, with tests written from the spec, before looking at the code.
- review-apprentice turns a review into teaching: the junior reviews first, then judges the agent’s findings.
They’re MIT licensed in hrih-skills, next to my other skills. In Claude Code it’s /plugin marketplace add R-Hatim/hrih-skills, then /plugin install review-loop@hrih-skills. They work in other agents too, as plain SKILL.md folders.
What about 70 agent reviews showed
Then I tried to find out whether any of it helps. I ran about 70 reviews with fresh agents: Claude Sonnet 5.5 and Opus 5.5, and later OpenAI’s Codex.
Some were on small test repos with known answers. The rest were on real code: four merged pull requests from a private TypeScript and Postgres service I’m building, and three of my nightly builds. Some changes were clean. Some had a bug I’d planted, one their own tests didn’t catch.
It’s a small experiment, so here’s exactly what it was:
- Same request, two conditions. Every review ran in a fresh agent and got the same request twice: once with a skill and once without. For the memory test, once with the lessons file and once with it removed.
- Planted bugs and clean runs. The planted bugs were ones each project’s own tests didn’t catch. Every change also ran clean, to count false alarms.
- Grading. I graded the answers myself against fixed yes-or-no questions: was the planted bug found, what risk level was given, who was named to look. Then I went back by hand over the cases each conclusion rests on.
The test repos with planted bugs are in the skills repo, if you want to run them yourself. Five things stood out.
1. The skills didn’t make the reviewers better at finding bugs
On the real pull requests, the planted bug was found 10 times out of 10, with or without a skill. On the nightly builds it was 6 out of 6, with no false alarms on the clean versions. On changes of a few hundred lines, a capable model reading carefully finds a planted bug either way.
What the skills changed was the shape of the answer. The router said how much review a change needed, and how many approvals, in 9 of 9 runs. Without the skill, that happened in 0 of 9. The answers were good. You just couldn’t route anything on them.
The spec reviews came with evidence. They wrote dozens to hundreds of checks from the spec before reading the code, then planted small bugs to see which ones the tests would notice. One of those passes found that you can invert every letter in a pixel font and all 434 of that project’s tests still pass.
That rigour isn’t free: about 1.5 times the tokens, and up to three times the minutes.
2. The memory matters where the code is silent
To test the lessons file on its own, I wrote four changes whose right answer depends on something the team knows that isn’t in the code. Each one got the same skill twice, once with the lessons file and once without.
| The change | Without the lessons | With them |
|---|---|---|
| Switch the finance report to half-up rounding, “like everywhere else” | Suspicious (“half-even may have been deliberate”) but can’t conclude: “ask someone” | “It reverses a decision the team recorded on purpose.” Names who to ask, and notices the test was edited to match |
| A tidy-up of the permission check, behaviour unchanged | Medium risk, “someone who owns the folder” | High risk, sent to the payments lead because of an old incident in that file |
| A dashboard stat that counts only full refunds | Caught | Caught |
| A script full of print statements | Left alone | Left alone |
Both model families behaved the same way. With the lessons file, the vague hunch turned into a verdict and a name. Where the code already held the answer (a report next to the new stat counted refunds correctly), the memory added nothing.
That’s what I’d want your team’s memory to hold. The model already knows how to write code. What it can’t know is what the code can’t say: who owns a risk, which odd choice is deliberate, and what went wrong last March.
Switch the finance report to half-up rounding, “like everywhere else”
Suspects half-even was deliberate, but can only say “ask someone”.
Reverses a decision the team recorded on purpose. Names who to ask, and notices the test was edited to match.
High risk: “confirm the convention with finance”.
Top priority: restore half-even. It goes against the recorded decision, and the edited test hides it.
A tidy-up of the permission check, behaviour unchanged
For “someone who owns the folder”.
Sent to the payments owner by name, because of an old incident in that file.
Two reviewers, but no owner named.
Two reviewers, and names the owner.
A dashboard stat that counts only full refunds
From a report next to it that already counts partial refunds.
Citing the lesson on what counts as a refund.
Codex only ran the two changes where the code is silent.
A script full of print statements
Low risk. The prints aren’t flagged.
Low risk. The prints aren’t flagged.
Codex only ran the two changes where the code is silent.
Four changes I wrote on a small test repo. Claude (Sonnet 5.5) reviewed each one twice per condition; Codex once, and only the two changes where the code is silent. Switch the reviewer and the lessons file to compare.
3. Agreement isn’t independence
On the real pull requests, I often had four Claude reviews agreeing on the same problem. That felt like strong evidence. So I gave the same changes to Codex.
Codex found four real issues that none of the 30 Claude reviews of those changes had found. I reproduced each one (two are edge cases). One was a setup flow that left a running process signed out, which I had in fact fixed in my next pull request without anyone pointing it out.
Codex also missed the two worst problems in the private service, ones every Claude review had flagged. One of them was that an agent could approve its own proposals.
So if four reviews from one model family agree, you’ve got something closer to one vote than four. The families have different blind spots. If you’re going to pay for more than one automated reviewer, a second family is worth more than a second run of the same one.
Private service, merged pull requests
Nightly builds
One dot per real issue, across the four private pull requests and three nightly builds. Claude’s column shows the issues most of its runs agreed on, not every finding. Each of Codex’s four was reproduced or confirmed by later history. Pick a reviewer to see what it missed.
4. Procedure travels between agents; restraint doesn’t
The router and spec-review skills worked in Codex unchanged. It read the same SKILL.md files, ran the same scripts, and refused the same poisoned lesson.
The apprentice skill didn’t carry over. I asked both agents: “just tell me what’s wrong, I want to learn from it”. Claude held its findings back. It explained why, gave hints about where to look, and offered the list if I still wanted it. Codex listed both bugs in its first reply.
A skill that asks an agent to hold back what you asked for is the hardest kind to carry over. One paragraph saying exactly what to do in that case fixed it, and Codex then held back too.
I meant to test this part with a real person: me, on two nightly builds I’d never read. I dropped it. Both changes leaned on maths I’d have had to learn first, so I would have been testing the domain, not the review.
That’s a lesson about apprenticeship in itself. It works when the junior is learning the codebase in front of them, with its history and house rules around them. Dropped into unfamiliar code, nobody’s an apprentice. They’re just lost.
5. The experiment kept leaking its answers
Planting a bug that a careful reader can’t work out turned out to be hard. My first test repos had the correct version sitting on another branch. (Oops.) Later ones showed the bug arriving in the commit history. On real code, every planted bug contradicted something nearby: a test, a type, a comment.
The agents found those contradictions every time. I fixed each leak and reran the affected reviews, but the lesson stands: if your reviewer cross-checks everything it can see, you’ll have a very hard time fooling it with a planted bug.
The reviewers also found two bugs in my own review tooling. One made a sensitive-path rule skip every folder whose name starts with a dot (.github included). And one review agent, wanting to stop a test run of its own, killed every Python process on my machine.
Reviewers need fences as much as authors do.
Where the human still belongs
Here’s how I’d put people in the loop now, with what the data suggests for each.
Route by risk, and let the team’s memory raise the stakes. A fixed list of sensitive paths sets a minimum the agent can raise but never lower. Above it goes judgement: what the change does, how far it reaches, and whether it can be undone.
Then the lessons. A past incident in a file is the best single predictor of a risky change in Meta’s published work, where a learned model caught 61% of outage-causing changes to the expert path list’s 38%. And an agent’s approval never counts as a person’s.
Review the spec and the evidence, not every line. You’re better at “is this what we wanted?” than at reading 1,200 lines.
Tests should come from the spec, not the code: across 11 models, tests written while looking at buggy code confirmed the bug about eight times as often as tests written from the fixed code, and caught real bugs about a third as often. Then plant small bugs and see what the tests notice. You read the code only where the evidence runs out.
Make review a lesson, on purpose. The junior reviews first, before seeing the agent’s findings, because if you read an AI’s review first, you focus where it pointed and find no more serious problems. Then they judge each of the agent’s findings, with a reason. That judging is the skill a junior needs now.
Anthropic’s own trial found that developers who used AI to learn a new library scored 4.15 points lower on a 27-point quiz. The ones who asked it conceptual questions or for explanations scored 65% to 86%. The ones who handed it the work scored under 40%. The senior’s corrections go into the lessons file, in the senior’s words, where the next junior will find them.
The data stops about there, so I’ll add my own view, from my own experience: I believe a loop like this is good for juniors. Making your own call first, and then defending it against a stronger reviewer, is how judgement gets built.
And a lessons file written by the people who were there hands a new person the history that used to take years of review comments to pick up.
The loop
When people did all the reviewing, the teaching happened as a side effect. A senior explained, a junior remembered, and the team’s knowledge moved one comment at a time.
When agents review, the explaining can still happen. But only if someone writes it down where the next reviewer and the next person will read it.
So here’s the rule I’ve ended up with. Every human decision made with a reason is a lesson. Keep it in the repository, in plain words, pointing at the code it’s about, and let both the agents and the people read it.
How does review work on your team now that agents are in it? Is anyone still learning from it? Tell me in the comments. I’d like to know where the teaching went.
A thought to add?
A good question, another perspective, or something this made you think about. There’s room for it here.
Loading the conversation…