A Second Review Needs a Different Job

An agent had finished a code change. A separately invoked agent, using a different model, reviewed the security boundary and found a route with no check on who was allowed to view it.

The builder had missed it completely. I wanted the second reviewer to show exactly where the check was missing.

The reviewer found a specific defect we could inspect. Changing models may not have caused the discovery. Two models can miss the same thing. I gave the second reviewer a different job.

Ask it to “review this again” and it may repeat the builder's reasoning back.

For an access-control review, I'd ask which routes require authorization, where each check is enforced, and what test shows that an unauthorized request is refused.

I wouldn't trust the review unless it points to the code and the test result. If it says every route is protected but can't show the check for one route, that route remains an open question.

I also want the reviewer to explain the permission it expected. The reviewer needs enough product context to distinguish an intentionally public route from a missing restriction. Without that context, the reviewer can flag a pile of things that don't matter.

I'd test the review process against a small set of resolved defects using isolated before-and-after copies. The earlier copy contains a problem the team has already established. The corrected copy shouldn't receive the same finding.

Miss the known defect, or complain about code that's already fixed. Either result tells me the review job needs work. I keep the expected findings out of the review prompt.

The copies need separate runs. Showing the fix beside the defect gives the reviewer the answer.

This test won't cover every future security mistake. It shows whether the second reviewer can do this specific job before I rely on it.

In that review, the output I cared about was the unprotected route, the missing check, and a test that could reproduce the problem.

Source: Correlated Errors in Large Language Models.