RWE-bench: on 20 real engineering tasks, the best AI model passes just 37% of its runs

On October 27, 2026, Xplore, a company that builds training environments for AI models, released RWE-bench (Real World Engineering Bench). The benchmark gives AI agents twenty design tasks in mechanical, electrical and aerospace engineering. A program checks every submission and runs physics solvers where needed. GPT-6 Astra (OpenAI) tops the leaderboard dated October 28, passing 37% of its runs, ahead of Claude Opus 5.5 (Anthropic, 29%) and GPT-6.1 Sol (OpenAI, 25%).

Five of the eleven models tested did not pass a single run. Four of the twenty tasks defeated every model across nearly 1,100 runs. “AI isn’t great at designing rockets, engines, or chips yet,” Xplore wrote on X when it introduced the benchmark.

Twenty tasks from real engineering work

The benchmark has nine mechanical tasks, five electrical and six aerospace. In mechanical engineering, the agent models engine parts, sheet-metal parts or printed mechanisms in CAD, and the program then checks their strength, fit and motion. In electrical engineering, the agent routes high-speed links (DDR memory, PCIe) on dense, already populated boards. In aerospace, it runs computational fluid dynamics (CFD) and heat-transfer simulations, from building the mesh to a converged solution, on turbine vanes, aircraft wings and rocket nozzles.

Xplore describes three of the tasks in detail. The first asks for a one-piece steel crankshaft for a three-cylinder diesel engine, with its drilled oil passages. The part has to fit the bearings, flywheel and front drive it is given without going over a mass limit. Under full load at 2,400 rpm, it has to stay within its stress, stiffness and bearing-load limits and last millions of cycles without fatigue failure.

The second is routing the DDR3L memory on a production FPGA board: a 32-bit bus from a Zynq-7000 to two memory chips. That means routing 71 signals out of a 484-ball BGA package on an eight-layer board, where nearly 300 parts are already placed and every other signal is already routed, without moving any of them. The third asks the agent to compute the heat transfer around a high-pressure turbine vane that sits right behind the combustor, where the gas is hotter than the metal can withstand.

According to Xplore, each task was written by an engineer on its staff who specializes in that field. That engineer also wrote the verification program and a reference solution that meets every requirement, using the same tools the agents get.

Graded by a program

Neither people nor language models grade the submissions. A deterministic program checks each design against every requirement in the specification. Missing a single requirement fails the run.

The agent gets the same specification a human engineer would. It works in a sandbox with professional CAD and simulation software for up to 24 hours. The average run took 3.4 hours across all models.

Each model works inside its own vendor’s agent: Codex for OpenAI, Claude Code for Anthropic, Gemini CLI for Google, Kimi Code for Moonshot AI. The benchmark page says each gets five runs per task at maximum reasoning effort.

The October 28 leaderboard

GPT-6 Astra passes an average of 37% of its runs across the twenty tasks, with a 95% credible interval of 30.7% to 43.8%. Claude Opus 5.5 follows at 29% (21.9% to 37.6%), then GPT-6.1 Sol at 25% (18.8% to 31.9%). The intervals of the top two overlap between 30.7% and 37.6%, so GPT-6 Astra’s lead over Claude Opus 5.5 is not settled. On November 3, OpenAI said two of its staff had used GPT-6 Astra to design a small device, from the case to the firmware, while noting that the model did not consistently account for the manufacturer’s tolerances.

Google’s Gemini 4 Argon comes in at 12%. Anthropic’s Claude Fable 5.1 manages only 7%, far behind Opus 5.5; our guide to Claude models explains the two families.

Five models scored zero: DeepSeek V4.1 Flash, Qwen3.8 Max (Alibaba), GLM-5.3 (Z.ai), Grok 4.7 (xAI) and Meta’s Muse Wonder.

Five attempts reshuffle the ranking

Xplore also publishes a pass rate over several attempts, in which a task counts as solved if any run passes. With five attempts, Claude Opus 5.5 passes 70% of the tasks and moves ahead of GPT-6 Astra (60%) and GPT-6.1 Sol (50%). Gemini 4 Argon climbs to 35% and Fable 5.1 to 20%.

“Capable, but not yet consistent,” the page says of Opus 5.5. Anthropic’s model solves less than a third of the tasks on the first try and more than two-thirds within five attempts.

The price of a run

For each model, Xplore reports the average API spend for one run on one task, in dollars. The amounts below are converted to euros at the European Central Bank’s reference rate for October 27, 2026 (€1 = $1.115).

GPT-6.1 Sol averages $10.28 in API spend per run, or €9.22. GPT-6 Astra ($58.83, or €52.76) and Claude Opus 5.5 ($57.52, or €51.59) cost almost six times as much, while scoring 12 and 4 points higher, respectively. “Spending more buys little,” reads the heading on Xplore’s page. Claude Fable 5.1 has the highest average cost on the leaderboard, $97.57 per run (€87.51), for a 7% pass rate.

Xplore says longer runs did not score better: GPT-6 Astra, the leader, is among the fastest. Opus 5.5 processed an average of 91.7 million input tokens per run, compared with 32.8 million for Astra.

GPT-6.1 Terra, which OpenAI presented on October 26 as costing one-ninth as much per task as Claude Opus 5.5 on one of its own tests, is not on this leaderboard.

Gemini 4 Carbon, the Argon successor Google unveiled on October 28, is not on it either: the model tops the Artificial Analysis index by one point over Claude Opus 5.5.

The benchmark’s limits

The benchmark has just twenty tasks, and only three are described in detail. Xplore does not say which four tasks no model passed, and the page offers neither the verification programs nor the reference solutions; access requires contacting the team. As of October 28, we had found no independent replication of the results.

The data file published with the page calls itself provisional: it comes from an October 25 export, made while some runs were still in progress, and the leaderboard counts 99 or 100 runs depending on the model. It also lists a “high” effort setting for Gemini 4 Argon and “xhigh” for Qwen3.8 Max and Grok 4.7, while the page states maximum effort for all of them. Neither source says whether those are the highest settings each tool offers. Because each model works inside its vendor’s agent, the score measures the model and that agent together.

The page gives no human baseline either. It says passing any one of these tasks takes an expert engineer trained in computer-aided engineering software, but it publishes no measured pass rate or time for engineers. A Reddit post on October 28 claimed that most people would pass none of them; the page does not measure that.

Xplore also sells training environments for models, built with the same method as the benchmark, and says the benchmark tasks themselves are never licensed for training. A lab can ask to have its model evaluated, and its score appears on the leaderboard only if the lab chooses.

On launch day, venture firm Lightspeed congratulated the team on X, and podcaster Dwarkesh Patel posted there that he was proud to be backing it. RuntimeWire, which covered the launch on October 27, noted that Xplore’s business depends on whether expert-built tasks can become useful training infrastructure.

What it means

Lightspeed noted in its post that physical engineering still lacks the evaluations and verifiable rewards that code and math have. RWE-bench grades engineering deliverables by computation and puts a price on every run. The first leaderboard is clear: inside their vendors’ commercial agents, the best models fail most of these tasks on the first try. With five attempts, though, the best one passes seven in ten.

For engineering firms, electronics engineers and independent engineers who follow these tools, the most telling result is the inconsistency: the same task can pass on one run and fail on the next, with the same model and the same tools. The benchmark also shows that a higher price buys few points: the other two models in the top three cost almost six times as much as the cheapest, for 12 and 4 more points.

These design failures contrast with an announcement from the same week: OpenAI said its AI had disproved the Exponential Time Hypothesis, a conjecture in theoretical computer science. In “The physical world is next,” a page on its website, Xplore describes the same gap with a different example: this year a model produced a proof on a Millennium Prize problem about how fluids flow, yet models still cannot reliably design a pipe network that actually carries water.

The numbers are still Xplore’s own: its tasks, its verification programs and a data version marked provisional. The company says harder tasks and more fields are coming, including biomechanical, civil and materials engineering. The next things to watch are a third-party replication, other labs joining the leaderboard and how the next models score.

Sources

• RWE-bench · Real World Engineering Bench, Xplore, results as of October 28, 2026
• RWE-bench leaderboard data file, Xplore, export of October 25, 2026
• Introducing Real World Engineering Bench (RWE-Bench), Xplore on X, October 27, 2026
• The physical world is next, Xplore, accessed October 28, 2026
• Xplore AI launches engineering benchmark, with GPT-6 Astra at 37%, RuntimeWire, October 27, 2026
• Post on the RWE-bench launch, Lightspeed on X, October 27, 2026
• Post backing Xplore, Dwarkesh Patel on X, October 27, 2026
• Euro foreign exchange reference rates: US dollar, European Central Bank, October 27, 2026

Header image: MACCUS chart, data from Xplore (RWE-bench, results as of October 28, 2026).