44.3% Python Pass Rate: JetBrains' CTO on Where Claude Fable 5 Shines

Hi, I'm Shii-chan! Today I want to share an interview from Claude Blog. It's about how a company that makes developer tools actually evaluates and uses the newest models in practice, and I found it really fascinating!

Claude Blog claude.com

What was announced?

Claude Blog published "Securing the frontier: How JetBrains evaluates and deploys Claude Fable 5," an interview with Vladislav Tankov, CTO at JetBrains, the company behind IntelliJ IDEA, PyCharm, and Kotlin. JetBrains serves more than 12.5 million active users and 88 of the Fortune Global 100. In the interview, Tankov talks about how his team evaluates new models, when they reach for Claude Fable 5, and how they think about safety and data retention.

Why it matters

Tankov has been at JetBrains for 10 years and says the company was among the very first customers of LLM providers. Over the past year, he says AI skeptics inside the company and among customers have essentially disappeared, calling it a foundational shift for the industry.

That's exactly why it matters how a coding company like JetBrains evaluates models. Some models score well on public benchmarks but fall short on real work, so JetBrains runs a large evaluation pipeline against private repositories, including its own monorepo, and tracks leaderboards for quality, cost per task, and speed.

What changes

• Claude Fable 5 posted a 44.3% Python pass rate in JetBrains's evaluation suite, versus 28.2% for Opus 4.8, a 16-point jump
• In head-to-head Python comparisons, Claude Fable 5 solved 18 tasks that Opus 4.8 missed and lost only 2
• When its code ran, Claude Fable 5 passed tests more often, making its answers more trustworthy
• It needed about 22% fewer steps than Opus 4.8 to reach a solution, meaning less trial and error

Dive Deep

Here are some of the technical details I found most interesting.

• On Java tasks, Opus 4.8 often tried pulling in outside resources that rarely helped, while Claude Fable 5 skipped that and worked with the code already in front of it, showing better engineering habits
• Opus is treated as the reliable workhorse, while Claude Fable 5 is the model JetBrains reaches for when it needs strong reasoning or feels like it needs a partner. One tech lead used it to implement a rich text editor component the team had attempted several times before, and Claude Fable 5 nearly one-shotted it
• Claude Fable 5 also powers long-running agentic-coding experiments, where an agent is given specs (text and images) to build sophisticated IDE-like apps, and can even generate its own specs from an existing app to rewrite it into a different runtime, framework, or language in a nearly black-box setup
• Security is a major use case: JetBrains runs white-box testing against its own products to find vulnerabilities with Claude Fable 5, while also preparing for outside actors probing with similar-class models
• On data retention, JetBrains would prefer zero retention but sees reviews limited to the most serious flagged cases as a fair tradeoff for understanding classifier errors
• JetBrains trusts Anthropic's red-teaming work on the model itself, and instead focuses its own effort on building the infrastructure and safety net around the model and harness

Wrap-up

• JetBrains CTO Vladislav Tankov shared how the company evaluates and uses Claude Fable 5 in an interview with Claude Blog
• Claude Fable 5 hit a 44.3% Python pass rate versus Opus 4.8's 28.2%, while needing about 22% fewer steps
• The team reaches for Claude Fable 5 for harder reasoning tasks and situations where the path forward isn't obvious
• The interview also covers concrete practices around security testing and data retention
• This should resonate with engineering leaders who want to know how a large dev-tools company evaluates and rolls out frontier models