How Cloudflare Cut Its AI Code Reviewer's Inference Bill by 68%

Hi, it's Shii-chan! Today's story from the Cloudflare Blog is a practical one about cutting AI costs.

Cloudflare Blog blog.cloudflare.com

What was announced?

The Cloudflare Blog published a case study on cutting the inference bill of its in-house AI code reviewer by 68%. With no major architectural changes, the cost had doubled, so the team dug in and reworked prompt caching and model routing.

Why it matters

It started when cost per review doubled:

• Average cost: about $1.19 in April to about $2.46 by late August
• Tokens per review: 0.91 million to 2.99 million
• The cache hit rate had risen from 85.7% to 97%, yet cost still went up

Before optimization, spend split roughly 23% output tokens, 43% cache writes, and 34% cache reads. Most of the money went to context the model had already seen or would see again. When all reviewers moved to Anthropic models in August, cost per job went from $1.38 to $2.03, and the standards reviewer on Claude Opus 4.7 went from about $0.25 to $0.88 per review.

What changes

The average cost per review fell from $2.43 to $0.77, down 68%. Of the $1.67 drop, code changes accounted for about $0.65 and model and effort changes for about $1.06.

Latency improved too:

• Median: 4m 09s down to 1m 32s
• 99th percentile: 15m 20s down to 4m 41s
• Reviews over five minutes: 37.9% down to 0.7%

Dive Deep

There were six changes:

• Share a cached prefix across repositories: The first request of each session hit the cache only 14 to 26% of the time. In OpenCode v1, the only cache breakpoint sat after repository-specific content (working directory, date). The team split the system prompt at a marker so the shared part (tools and the agent prompt) got its own breakpoint, and removed repo-specific content like the skills list from it. First-request cache reads rose to 91 to 99.5%
• Preload reference material: The standards reviewer had spent its first turns reading reference files via tool calls. Putting that index into the prompt cut LLM calls per run from 15.9 to 9.6 and cost per run from $1.12 to $0.72
• Reduce calls per session: Removing tools sub-reviewers could not use, reading files in parallel (64% of turns had made just one tool call), skipping reviewers with nothing new on re-reviews, ending the session after the review is posted, and stopping canceled jobs (checked every 15 seconds). Together these saved about $0.20 per review
• Share one prefix among three sub-reviewers: The code quality, security, and performance reviewers send identical first requests. One writes the cache and the other two wait up to 20 seconds to read it, saving about nine cents each
• Tune models and effort: Underperforming model and effort combinations were dropped, and the coordinator moved to Opus 5.5 at medium effort. In an A/B test of Sonnet 5 vs. Sonnet 5.5, Sonnet 5.5 was 47 to 62% cheaper per run with similar findings. Code quality moved to high effort on 5.5 after testing
• Move from OpenCode v1 to v2: A shadow comparison showed 98% agreement on approval decisions, and full-tier reviews finished about 13% faster

Quality checks (code quality, high effort):

• Critical and warning findings fell 40%, but authors acted on 59% of serious findings, up from 34%
• In blinded judging, important findings per review were stable (0.28 before, 0.30 now)
• Severity inflation dropped about 70%, mostly from fewer minor findings being rated critical
• Reviews without a verdict fell from 0.79% to 0.27%, and security reviewer retries fell from 3.5% to 0.04%

The post is honest about caveats:

• Comparisons use weekday windows, not controlled experiments
• "Authors acted on the finding" is a proxy for action, not correctness
• Judges were also models, and some groups had only 35 to 66 findings
• The high-effort test covered only one day
• Medium effort lost real findings for code quality and performance (about 64% fewer important performance issues per review). Security may have lost some too, though not significantly. Security and performance stay at medium effort, with high-effort testing planned

Wrap-up

• Costs doubled because re-reading context (cache writes and reads) made up most of the spend
• Revisiting cache breakpoints and sharing common prefixes helped a lot
• Preloading reference material and cutting tool calls reduced LLM calls themselves
• Models and effort were re-chosen while measuring quality
• The result: 68% lower average cost and a median latency drop from 4m 09s to 1m 32s

If you run LLM-based agents and want lower cost and latency, this is a great read for you!