Claude Sonnet 5, which Anthropic released on June 30, 2026, passed 4 of the 70 tasks in Terminal-Bench-Science on Vals AI’s tests. Claude Sonnet 5.5, released on September 28 at the same token price, passed 33. Each task is a piece of research work written by a working scientist and graded by tests that scientist wrote. Vals updated the leaderboard on October 14, then on October 16 folded the benchmark into a new composite score, the Vals Science Index.
Models from OpenAI and Google made the same kind of jump. OpenAI’s Sol line went from 14 tasks with GPT-5.6 Sol, released to the public in July, to 37 with GPT-6.1 Sol, released September 29. Google went from 1 task with the February preview of Gemini 3.1 Pro to 31 with Gemini 4 Argon.
On the new index, GPT-6.1 Sol scores 59.63% for $1.36 per test. Claude Sonnet 5.5 scores 59.12%, half a point lower, and spends $13.24.
One index built from three benchmarks
Vals is an independent AI evaluation company. Its Science Index is the plain average of three agentic benchmarks, each of which gives a model a dataset, a simulated lab or a terminal and scores what it finds.
BioMysteryBench, built by Anthropic and described in April, asks 99 bioinformatics questions about real datasets. Up to five domain experts attempted each question, and the 23 that none of them solved form a “human-difficult” set.
MysteryMechanism, built by Vals, hides 222 mathematical laws drawn from biology, physics, chemistry, engineering, ecology and abstract dynamical systems. The agent gets two noisy observations and a small experiment budget, with no internet access and no hint of the domain, and must submit one executable law.
Terminal-Bench-Science, by Steven Dillmann and the Terminal Bench team, allows up to eight hours of agent time per task, twice BioMysteryBench’s four-hour limit. Its 70 tasks cover the life sciences (19), the physical and mathematical sciences (17 each), engineering (9) and the earth sciences (8). Each author’s tests run in a sandbox the agent cannot reach, and there is no partial credit. In Vals’ words, “a nearly right research result is unusable.”
Four model lines, before and after
On Terminal-Bench-Science, Vals scores each model on a single run, using the same general-purpose agent harness, mini-swe-agent. The table compares the latest model in each line with an earlier one.
| Earlier model | Later model | Tasks passed (of 70) |
|---|---|---|
| Claude Sonnet 5 (June 30) | Claude Sonnet 5.5 (September 28) | 4 to 33 |
| Claude Opus 5 (July 24) | Claude Opus 5.5 (September 22) | 19 to 33 |
| GPT-5.6 Sol (July) | GPT-6.1 Sol (September 29) | 14 to 37 |
| Gemini 3.1 Pro Preview (February 19) | Gemini 4 Argon (September 30) | 1 to 31 |
Anthropic’s Sonnet and Opus lines ended at the same score, 33 tasks; our guide to the Claude models explains how the two differ. OpenAI’s intermediate model, GPT-6 Sol, passed 21. The leader is OpenAI’s flagship, GPT-6 Astra, with 44 tasks, or 62.86%.
The labs’ own runs, made on their own setups, show the same jump at higher scores, as Vals expects. Anthropic’s Opus 5.5 announcement reports 58.7% on Terminal-Bench-Science 0.1, against 29.0% for Opus 5. It gives a standard error of 3.5 to 5 points per model.
OpenAI says GPT-6.1 Sol “more than doubles GPT-6 Sol’s score” at maximum reasoning effort, and that GPT-6 Astra still scores highest among the models it tested, at 68.1%. Neither company’s numbers can be compared row by row with the Vals table.
The price of a test run
Claude Sonnet 5 and 5.5 share a list price of $2 per million input tokens and $10 per million output tokens. Vals measured nearly the same cost per Terminal-Bench-Science test for both, $30.30 and $32.52, so the eightfold gain came at roughly flat cost. OpenAI’s Sol line got cheaper as it improved: GPT-5.6 Sol cost $7.32 per test on Vals’ run, GPT-6.1 Sol $3.44. Google’s line moved the other way, from $5.91 per test for the February preview to $44.90 for Gemini 4 Argon, the most expensive of the four latest models.
GPT-6.1 Sol and Claude Sonnet 5.5 also share a list price, yet on the index Sonnet 5.5 costs about ten times as much per test. Over the index run, it consumed 20.05 billion input tokens, against 757 million for GPT-6.1 Sol. OpenAI’s own figures for Terminal-Bench-Science 0.1, at maximum effort, point the same way. They put GPT-6.1 Sol at $5.47 per task, against $23.21 for Opus 5.5 and $23.80 for GPT-6 Astra.
Epoch AI looked at the field as a whole in an analysis published September 22. In “The plunging price of thought”, Luke Emberson and David Roodman estimate that the cost of reaching a fixed level of AI performance has fallen about 47% per quarter since 2023, or 13x per year. By their measure, no technology they compared it with got cheaper as fast. The decline is four times as fast as the fall in DNA sequencing costs, and 54 times as fast as the fall in electricity prices in the century before 1973.
For performance that has just become state of the art, the drop is 66% per quarter. The estimate averages five benchmarks in mathematics, hard science and games of skill over many models, so it cannot be read as the price curve of any single model named here.
What the scores leave out
All of Vals’ Terminal-Bench-Science scores come from one run per model; BioMysteryBench, a third of the index, averages three runs. On 70 tasks, a single task moves a Terminal-Bench-Science score by 1.43 points. On the index, Vals shows a margin of about 2.3 points on each of the top scores, so GPT-6.1 Sol and Claude Sonnet 5.5, half a point apart, are statistically level.
Vals re-attempted tasks that ended in an infrastructure error and says a handful were solved on such a retry. The board is also moving from the Terminus 2 harness to mini-swe-agent, and Vals expects agents tuned by each vendor to score higher than its general-purpose one.
Refusals count as failures: on BioMysteryBench, biology content filters blocked some tasks for several models, and Claude Sonnet 5.5 refused the same three tasks in each of its three runs. Anthropic’s April study also found that many wins on the human-difficult questions are hard to repeat. For Claude Opus 4.6, 44% of the hard problems it solved were solved only once or twice in five attempts, against 9% on the questions experts could answer. Anthropic wrote that such wins are often “lucky reasoning paths rather than reliably reproducible solutions.”
The tasks cover one kind of work: a well-posed problem whose answer a program can check. Vals’ announcement calls the index “a measure of AI’s ability to make scientific discoveries,” while what it scores is whether an agent’s output passes an automatic check. Choosing which question to ask is outside the test. On the part that is tested, the top model, GPT-6 Astra, still fails more than a third of the index. On October 16, the day the index came out, physicist Emre Onur Kahya published an audit, run with Claude, of his own 2007 quantum gravity calculation, which found a sign error (see our article on the audit).
What it means
On Terminal-Bench-Science, four model lines from three labs went from passing between 1 and 19 tasks to passing between 31 and 37 in their latest release, close to half of the 70. Cost has not moved as evenly as the scores. GPT-6.1 Sol, second on the index, runs it for about a tenth of what third-placed Claude Sonnet 5.5 spends, while Gemini 4 Argon costs more per test than Google’s February preview.
For research teams, the tasks in question are steps such as reconstructing a signal, calibrating a model, reducing an instrument’s output or proving a bound, the examples Vals gives. The five leading models on Terminal-Bench-Science now pass between 44% and 63% of them, and GPT-6.1 Sol does so for $3.44 per test on Vals’ run.
The pattern matches what we reported on AI-found counterexamples to a 1967 fusion conjecture and on Claude’s quantum circuit designs: AI moves fastest where a computer can check the result. All three benchmarks in the index are scored automatically, by the task authors’ tests, by hidden probes or, on BioMysteryBench, by an AI judge.
The next numbers to watch will come from the official Terminal-Bench-Science leaderboard, which averages three trials per task and pairs each model with its vendor’s own agent, and from Vals’ runs of the next model generation. On BioMysteryBench, the open question is whether any model starts solving the human-difficult questions on most of its attempts.
Sources
- Vals Science Index Leaderboard and Methodology, Vals AI, October 16, 2026
- Terminal-Bench-Science Leaderboard and Methodology, Vals AI, October 14, 2026
- BioMysteryBench Leaderboard and Methodology, Vals AI, October 12, 2026
- Evaluating Claude with BioMysteryBench, Anthropic, April 29, 2026
- Introducing Claude Sonnet 5, Anthropic, June 30, 2026
- Introducing Claude Sonnet 5.5, Anthropic, September 28, 2026
- Introducing Claude Opus 5.5, Anthropic, September 22, 2026
- The plunging price of thought, Epoch AI, September 22, 2026
Header image: chart by MACCUS, data from Vals AI (Terminal-Bench-Science leaderboard, October 14, 2026)
