On October 19, 2026, OpenAI said 86.6% of the COBOL rewrites by its newest mid-priced model, GPT-6.1 Sol, passed every hidden test in its evaluation, against 5.4% for the previous-generation GPT-5.6 Sol at the same low reasoning setting. COBOL is the 1960s language that, by IBM’s estimate, still handles most ATM transactions, and the test asks models to rewrite COBOL programs in modern languages. Given more time to reason, GPT-6.1 Sol reached 97.9%, which OpenAI’s post says brings the test “close to saturation.”
The scores come from an evaluation OpenAI built and has not released, but the model is public. GPT-6.1 Sol launched at OpenAI’s DevDay on September 29 and is available in the API as gpt-6.1-sol, at $2 per million input tokens and $10 per million output tokens, one fifth of GPT-6 Astra’s list prices. Our guide to OpenAI’s GPT models explains how Astra, Sol and Luna differ. OpenAI says the new model ran the COBOL test for 61% of GPT-5.6 Sol’s API cost at the same setting.
The code under card payments
COBOL was designed by CODASYL, a group of US government and business organizations formed in 1959, and its first version came out in 1960. According to IBM, it is still the foundation of more than 40% of online banking systems, supports 80% of in-person credit card transactions and handles 95% of ATM transactions. Those estimates have circulated for years, and OpenAI’s post repeats them with a link to IBM.
In a February 23 post, Anthropic wrote that “hundreds of billions of lines of COBOL run in production every day” and that the developers who built these systems “retired years ago.” COBOL is taught at “only a handful of universities,” the company added, and finding engineers who can read it “gets harder every quarter.” With AI, it argued, teams could modernize such systems “in quarters instead of years.”
IBM, which sells the mainframes much of that code runs on, saw its shares close down nearly 13.2% that day, CNBC reported. Reuters called it the stock’s steepest one-day drop since October 2000.
How OpenAI scored a rewrite
OpenAI’s test asks a model to migrate COBOL programs to C++, Java, Kotlin, Python or Rust without changing what each program does. Hidden tests then compare the new program’s observable outputs with the expected ones, byte for byte, and a migration counts only if every test passes.
That bar targets the way COBOL rewrites go wrong. Programs that move money depend on details such as rounding rules and the handling of missing records. OpenAI gives an example: COBOL’s fixed-point arithmetic drops extra decimal digits unless the code explicitly asks for rounding, so a rewrite that rounds “can run without errors while producing different financial results.”
A chart in OpenAI’s post gives the scores at five reasoning settings, from low to maximum. At the maximum, GPT-5.6 Sol reached 65.2% and GPT-6.1 Sol 97.9%, so the gap narrows as both models reason longer. Even at low effort, GPT-6.1 Sol beat GPT-5.6 Sol at maximum effort, at about a seventh of the cost per graded attempt ($0.31 versus $2.06).
Most of the low-effort gain came with GPT-6.1 Sol itself. GPT-6 Sol, released a week earlier on September 22, scored 20.8% at low effort and 95.8% at maximum effort on the same test.
Six months ago, agents failed without noticing
In April, the AI coding company Factory published Legacy-Bench, a set of maintenance tasks in COBOL, Fortran, Java 7 and other older languages. Across the 12 model and agent combinations it tested, overall pass rates ran from 16.9% to 42.5%. Of the 44 tasks no model solved, 31 were in COBOL, and Factory wrote that migrating COBOL to Java, Rust or C++ “rarely works.”
The failures were also hard to see. COBOL bugs are silent, according to Factory: in one payroll task, an agent computed a health deduction of $275 instead of $125, the totals came out wrong, and 24 of 29 tests still passed. “In 97% of failures, the agent believes it has solved the task,” the company wrote.
The two sets of numbers come from different tests and different models (Factory’s top performer in April was GPT-5.3-Codex), so they don’t line up point for point. Both describe the same failure, code that compiles and runs but gives wrong results, which is what OpenAI’s byte-for-byte check is meant to catch.
A database engine in 20 languages
OpenAI’s second test, MultiLangBench, is public on GitHub under the Apache 2.0 license. It asks a model to build a database engine compatible with SQLite 3.40.1 in an assigned programming language, working only from SQLite’s documentation, with no internet access and no copy of SQLite left in its environment. A solution in the wrong language scores zero. The tests come from SQLite’s own suite: 24,000 of its 51,000 tests could be converted, and the repository tells evaluators to use a 272-test subset.
At maximum reasoning effort, GPT-6 Astra, OpenAI’s top model, passed 87.4% of tests on average across 20 languages, compared with 26.8% for GPT-5.6 Sol, and scored above 83% in 19 of the 20. COBOL stayed the hardest, at 72.1% (10.9% for GPT-5.6 Sol). Fortran rose from 21.4% to 83.5%, Visual Basic from 27.5% to 89.4% and OCaml from 25.6% to 85.8%. The cheaper GPT-6 Luna reached 40.3% at an estimated $0.77 per attempt, versus $3.48 for GPT-5.6 Sol.
OpenAI’s text compares its top model with an older mid-priced one, but a chart in the same post shows the mid-priced GPT-6.1 Sol close behind Astra, at 87.0% on average. Large programs built mostly by AI are already shipping outside labs: on October 8, an open-source team released free Word, Excel and PowerPoint alternatives written largely with Claude.
The limits of OpenAI’s tests
The COBOL and MultiLangBench scores are OpenAI’s, measured on its own tests. The COBOL test is private: OpenAI has not said how many programs it contains, how long they are or what kind of business logic they cover, and no one outside the company can rerun it. Passing every hidden test means matching outputs on the inputs those tests cover, and says nothing about inputs they never tried.
OpenAI does not explain how it trained its models for these languages, only that the work followed feedback from developers and enterprise customers that coding models “often struggle” with them. For production software, OpenAI writes, applying the capability “still requires expert review and testing against the system’s business rules and dependencies.” The Japanese tech site TechnoEdge, which covered the post on October 20, made the same point: no one runs a production system on 86.6% or 97.9%, so people still have to check the work.
What it means
Six months ago, the best coding agents in Factory’s tests failed at most COBOL work and, when they failed, nearly always believed they had succeeded. On OpenAI’s own test, a public mid-priced model now rewrites COBOL programs with matching outputs almost nine times out of ten at its cheapest setting, better than the previous generation managed at its most expensive one. Banks, insurers and governments have long put off rewriting these systems. Part of the reason, Anthropic wrote in February, was that understanding legacy code cost more than rewriting it.
The bottleneck shifts to verification. OpenAI’s result rests on hidden tests that pin down what each old program does, and a real migration needs the same kind of tests at far larger scale, written by people who know the business rules. Another field where output can be checked against an original has already been through this: fans who spent years rebuilding the source code of video games saw AI models speed the work up so much that their tracking hub is closing.
The next evidence will come from outside OpenAI. Anyone can now run MultiLangBench, so independent scores should follow. The COBOL migration test is still private, and Factory has not published Legacy-Bench results for this generation of models. A bank or government agency that publishes results from a migration done this way would provide the first data from production.
Sources
- Advancing coding in low-resource programming languages, OpenAI, October 19, 2026
- openai/MultiLangBench, OpenAI on GitHub, October 19, 2026
- Introducing GPT-6.1 Sol, OpenAI, September 29, 2026
- Legacy-Bench: Can AI Agents Maintain the World’s Most Critical Software?, Factory, April 1, 2026
- COBOL Modernization with AI: Breaking the Cost Barrier, Anthropic, February 23, 2026
- What Is COBOL?, IBM, May 8, 2025
- IBM is the latest AI casualty. Shares tank 13% on Anthropic programming language threat, CNBC, February 23, 2026
- COBOLやVBなど旧式言語でGPT-6急伸、正解率5%から最大98%とOpenAI発表, TechnoEdge, October 20, 2026
Header image: chart by MACCUS, data from OpenAI (October 19, 2026)
