On October 22, 2026, Gary King and Steven Worthington of Harvard’s Institute for Quantitative Social Science posted a working paper that pits AI-written peer reviews against human reviews of unusually high quality. Three former editors of leading journals, who were not told that AI was involved, compared 10 randomly drawn pairs of reviews. In 26 of their 30 judgments, they preferred the review written by the model.
The reviewer was Claude Opus 4.8, which Anthropic released on May 28; Opus 5 and Opus 5.5 have followed. An AI judge from another company, OpenAI’s GPT-5.5, preferred the AI review in all 132 pairs it could decide. Checked against the manuscripts, 5.5% of the AI reviews’ claims were judged incorrect, against 15.9% for the human reviews, and the AI reviews cited no fabricated sources.
The study is a working paper and has not been through peer review itself. It arrives as journals and conferences struggle to find reviewers for a fast-growing number of papers.
A review system under strain
AAAI, a large artificial intelligence conference, received more than 30,000 initial submissions for its 2026 edition, up from about 15,000 a year earlier, according to an April paper by the team that ran its AI review pilot. It had to recruit more than 28,000 reviewers and chairs, nearly three times the previous committee. ICML 2026, a machine learning conference, handled over 24,000 papers and 17,000 reviewers.
The Economist described the same pressure on October 23 in its Off the Charts newsletter: AI “has also made it easier to produce low-quality and even fraudulent science, which is overwhelming systems of moderation and peer review.” The piece quotes James Wilsdon of University College London on an Ivy League researcher who once spent a day a week on grant proposals and now spends 45 minutes. Papers are coming faster as well: a Harvard physicist produced 36 manuscripts in three months with Claude, and in October three teams posted AI-assisted proofs of the same 31-year-old geometry conjecture.
How the Harvard test worked
The human reviews came from rapidPeer, a service built by the Harvard institute and since expanded to other institutions. It gives university faculty double-blind reviews of their drafts before they submit to a journal. Reviewers, chosen by experts in the field from other institutions, are asked to write at least 1.5 to 2 pages and are “paid (well),” in the authors’ words. In a survey a week after delivery, 88% of 74 authors rated their review high or very high quality.
The corpus is every eligible rapidPeer submission: 124 papers and 134 human reviews collected between 2019 and 2026, from statistical analyses to formal mathematical models and purely qualitative studies.
For each paper, Claude Opus 4.8 at its maximum reasoning setting wrote one review under two conditions. The “matched” version received the manuscript and the same reviewer guidelines as the humans, with no tools. The “tuned” version got a purpose-built agent prompt and web search, which it used 12.4 times per review on average. The authors say the tuned prompt was written before any judging and never adjusted to the judges’ scores.
GPT-5.5 judged every pair twice, once in each presentation order, and a win counted only when it picked the same review both times. The three editors, given the same ranking instructions, agreed with each other 80% of the time, and with the AI judge 87% of the time on average.
The checks behind the headline numbers
To test whether the AI won by writing more, the authors kept only the 44 pairs where both reviews had similar lengths: the AI review won all 44. The AI’s lead was the same for papers submitted in 2025 and 2026 as for papers from 2022 or earlier, which suggests its advantage does not come from later work it may have seen in training.
In the main test, the AI won between 92% and 100% of decided pairs in each of the seven review areas, from methods to figures and tables. Sources were the one area where the matched AI, which had no web access, won only 62% of pairs; with browsing, it won all of them. In a replication with a different prompt and rubric, and with Claude Opus and GPT as both reviewers and judges, the AI review was preferred 97.1% of the time under the Opus judge and 94.2% under the GPT judge, across 416 comparisons.
The judge itself was tested on sentences planted in copies of each review, one accurate and one contradicting the manuscript: it flagged all 258 false sentences and none of the 258 accurate ones. Models of this kind are also being pointed at published work: on October 16, a physicist published a Claude re-derivation of his PhD calculation that found a sign error in a result printed in 2007.
Already running at conference scale
At AAAI-26, every one of the 22,977 papers sent for full review received one clearly labeled AI review, all generated “in less than 24 hours” for “less than $1 per paper,” paid with API credits OpenAI donated as a conference sponsor. That system was the pilot team’s own, built on an OpenAI model. Its reviews sat alongside at least two human reviews per paper, carried no score or recommendation, and replaced no human reviewer. In a survey with 5,834 responses, participants preferred the AI reviews on six of nine criteria, including identifying technical errors.
Many reviewers already use the tools, sometimes against the rules. At ICML 2026, where some reviewers were randomly assigned to a ban on language models, 22.5% of surveyed reviewers under the ban reported using one anyway. In a Frontiers survey of about 1,600 academics, reported by Nature in December 2025, more than half had used AI while peer reviewing.
What the study does not show
The authors name their main limit: “We did not measure the degree to which subsequent revisions improved the paper or the collective consequences of switching to AI reviewing.” The paper says its results concern the help a review gives authors and leave aside how editors should decide what to publish. The human-judge check rests on three editors and 10 papers.
King and Worthington also warn that AI tends to push research toward “the conventional, orthodox, and majoritarian,” and that AI reviews could make science more uniform. Earlier comparisons cited by the authors disagree with each other: on 119 BMJ Open articles, human reports raised more detailed and varied points than ChatGPT’s, while two editors at a cardiology journal rated AI reviews higher on five of seven criteria for 40 manuscripts.
A position paper by Joachim Baumann, Jiaxin Pei, Sanmi Koyejo and Dirk Hovy, accepted as an oral at ICML 2026, argues that “today’s AI systems should not be used to produce paper reviews.” On ICLR 2026 reviews, they found “a hivemind effect of excessive agreement,” and showed that “paper laundering,” having a language model rewrite a paper, can raise its scores from AI reviewers. AAAI’s survey respondents also rated the AI reviews more likely to overemphasize minor issues.
What it means
AI has sped up the writing of science more than its reviewing: papers and grant proposals take less time to produce, while a journal verdict still takes months. This study used human reviews selected for their quality, and the model came out ahead on every measure the authors built, with about a third of the human error rate.
The authors’ policy menu keeps the editor in charge: journals could publish an official AI review prompt, ask authors to submit an AI review with a letter answering it, and have editors run a second prompt they do not disclose, with manipulation checks. Some version of these ideas, they write, could enable “publication decisions within days rather than months,” while whether a correct paper meets a journal’s standards “can only be decided by a human journal editor.”
The first thing to watch is replication in other research areas, which the authors themselves call for, with larger panels of human judges. The second is evidence that papers revised with AI reviews end up better, which this study did not measure. The third is gaming. King and Worthington note that authors who revise for a published prompt may end up optimizing for it, and Baumann and his coauthors showed that a simple language-model rewrite of a paper can raise its scores from AI reviewers.
Sources
- AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices, Gary King and Steven Worthington, Harvard IQSS, October 22, 2026
- AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot, J. Biswas et al., arXiv, April 15, 2026
- Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026, S. S. Y. Kim et al., arXiv, September 16, 2026
- Stop Automating Peer Review Without Rigorous Evaluation, J. Baumann, J. Pei, S. Koyejo and D. Hovy, arXiv (ICML 2026), May 4, 2026, revised July 5
- Could AI help scientists with their paperwork, The Economist, Off the Charts, October 23, 2026
- More than half of researchers now use AI for peer review, often against guidance, Miryam Naddaf, Nature, December 15, 2025
- Introducing Claude Opus 4.8, Anthropic, May 28, 2026
Header image: chart by MACCUS, data from King and Worthington, Harvard IQSS working paper (October 22, 2026)
