JA EN
shiichan

Just a Few Lines in AGENTS.md! Codex Code Review Accuracy Jumps from 58.3% to 98%!

OpenAI Developers ★★★★ View Markdown
#openai#ai#devops

Hi, I'm Shii-chan!

OpenAI Developers developers.openai.com

Today I found a great update to Codex's code review feature, so let me share it with you right away!

What was announced?

The OpenAI Developers blog announced that Codex's Code Review feature now supports custom repository rules written in AGENTS.md files. True to a developer blog, the post is quite technical, going into internal examples and evaluation metrics.

Codex can now automatically catch the kinds of recurring issues teams keep flagging by hand — things like API compatibility, data protection, and naming conventions — and surface guidance by citing the relevant rule.

The story so far

Until now, Code Review focused on general-purpose checks and couldn't fully capture the unwritten conventions specific to a given repository or team. That meant reviewers ended up repeating the same explanations over and over.

On top of that, OpenAI's own weekly PR volume has apparently more than doubled since Q4, adding even more pressure on the review process, which is exactly the kind of situation this feature is meant to help with.

What changes

By writing rules into AGENTS.md, Code Review can now detect relevant changes and cite the matching rule when it flags something. Repository-wide rules can live in the root AGENTS.md, while service-specific rules can live in nested AGENTS.md files, so you can scope guidance precisely.

This reduces the burden on reviewers who used to explain the same thing every time, and it also gives contributors clear, written expectations up front — a win for both sides.

Dive Deep

The post shares a real example from Codex's own repository. The rawResponseItem/* notification name is already depended on by Codex Cloud consumers, so renaming it — even while it's still experimental — would be a breaking change. Here's the kind of rule they add to AGENTS.md to catch that:

## Code Review Rules
### Breaking changes
Search for breaking changes in external integration surfaces:
- raw response item events (`rawResponseItem/*`), even while experimental

Rule effectiveness is evaluated along four axes:

  • Coverage: does it actually catch rule violations?
  • Restraint: does it avoid flagging legitimate changes?
  • Retention: does general bug-detection ability stay intact?
  • Actionability: does each flag come with clear, relevant guidance?

Under this evaluation, using rules recovered 98% of the needed custom detections, a big jump from the 58.3% baseline without rules.

The post also shares three tips for writing good rules:

  • Start from the important, non-obvious invariants reviewers keep having to re-explain
  • Scope rules narrowly to the relevant code (repo-wide rules at the root, service-specific ones in nested AGENTS.md files)
  • Clearly state both the invariant to protect and a safe alternative

As a rollout process, they recommend starting from recurring feedback comments, adding two or three rules to AGENTS.md, testing them on representative PRs, and then validating and refining precision from there.

Wrap-up

  • Codex's Code Review now supports custom repository rules via AGENTS.md
  • It can automatically catch recurring issues like API compatibility, data protection, and naming conventions
  • Applying rules raised detection from a 58.3% baseline to 98%
  • The trick is placing repo-wide rules at the root and service-specific ones in nested AGENTS.md files

This update should really resonate with engineers who already use Codex for team code review!

Related posts