JA EN
shiichan

COBOL Migration Accuracy Jumps from 5.4% to 86.6%: OpenAI Levels Up Low-Resource Languages!

OpenAI News ★★★ View Markdown
#openai#ai#science

Hi everyone, it's Shii-chan! Today, from OpenAI News, a story about languages that rarely get the spotlight but matter a lot.

OpenAI News openai.com

What was announced?

OpenAI News announced stronger capabilities for building and migrating software in low-resource programming languages such as COBOL, Fortran, and Visual Basic.

Low-resource languages are ones with less code and documentation to learn from than Python or JavaScript. To measure progress, OpenAI built two evaluations:

  • MultiLangBench: tests whether a model can build a SQLite-compatible database engine from scratch across 20 programming languages
  • COBOL migration evaluation: tests whether a model can migrate existing COBOL programs to another language while preserving their behavior

Why it matters

COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions (figures OpenAI cites from IBM). Fortran is used in weather forecasting and nuclear reactor safety analysis.

Documentation may be incomplete, original developers may have left, and business rules may only exist in code, so maintenance and migration are hard.

What changes

Here are the results:

  • MultiLangBench: GPT-6 Astra reaches an average test pass rate of 87.4% (GPT-5.6 Sol: 26.8%), with improvements in every language tested
  • GPT-6 Luna reaches 40.3%, beating GPT-5.6 Sol at approximately 78% lower estimated API cost
  • COBOL migration: GPT-6.1 Sol hits 86.6% at low reasoning effort (GPT-5.6 Sol: 5.4%) at 39% lower cost. At higher effort it reaches 97.9%, nearly saturating the evaluation

Dive Deep

MultiLangBench uses database files as a shared interface. The candidate program gets an initial database and a SQL query, then produces a modified database. The reference SQLite implementation checks the result, so the same tests work regardless of language. The tests are adapted from SQLite's existing test suite.

Astra's per-language results at maximum reasoning effort:

  • COBOL: 10.9% to 72.1%
  • OCaml: 25.6% to 85.8%
  • Fortran: 21.4% to 83.5%
  • Visual Basic: 27.5% to 89.4%

Astra scores above 83% in 19 of 20 languages, with COBOL the hardest at 72%. Luna's estimated average API cost is $0.77 per sample versus $3.48 for GPT-5.6 Sol. The MultiLangBench setup and test suite are released on GitHub.

The COBOL migration evaluation targets C++, Java, Kotlin, Python, or Rust, and uses hidden tests requiring byte-for-byte parity on observable outputs. A migration succeeds only if all tests pass. For example, COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is requested, so a rewrite that rounds can run fine but produce different financial results.

OpenAI also notes that production use still requires expert review and testing against the system's business rules.

Wrap-up

  • Big gains in building and migrating software in low-resource languages
  • On MultiLangBench, Astra goes from 26.8% to 87.4%, and Luna reaches 40.3% at about 78% lower cost
  • COBOL migration goes from 5.4% to 86.6%, and 97.9% at higher effort
  • MultiLangBench is public on GitHub

This is for teams maintaining or migrating legacy systems!

Related posts