An AI paper checker audited 2,500 published machine-learning studies and human experts confirmed 263 of the 316 potential mistakes they inspected. The errors had survived publication at three influential research venues, including incorrect formulas, flawed derivations, contradictory explanations and figures that did not match the text.
The result comes from a Stanford-led preprint called To Err Is Human. Its GPT-5 system flagged an average of 4.66 possible objective mistakes per paper. In the validation sample, 83.2 per cent of the flags were genuine and 86 were potentially substantive.
That does not prove nearly every published paper is wrong. The AI flagged at least one issue in 99.2 per cent of the collection, but humans did not verify every flag. Most confirmed mistakes were minor, and the system missed 40 per cent of deliberately inserted errors. The breakthrough is narrower: AI can search for certain technical faults at scale, then hand suspicious passages to experts.
AI Paper Checker: What This Means for Science
Scientific publishing has always depended on limited human attention. Authors check their own work, reviewers assess it under deadline pressure and editors decide what deserves publication. A mathematical sign, table total or cross-reference can escape all three without anybody acting dishonestly or carelessly.
An automated checker changes the economics of that search. The Stanford team says its pipeline cost less than US$0.50 per paper. At that price, a conference could screen thousands of submissions for objective inconsistencies before asking human reviewers to spend their time on novelty, significance and whether the research question matters.
This is a practical example of the scalable oversight problem LiveAIWire has examined: humans increasingly need machines to help verify work that machines helped produce. The sensible model is not AI replacing peer review. It is AI performing a relentless first pass, with identified faults treated as allegations until a qualified person checks them.
The System Used One AI to Check Another
The researchers sampled 1,600 ICLR papers published between 2018 and 2025, 500 NeurIPS papers from 2021 to 2025 and 400 papers from Transactions on Machine Learning Research from 2022 to 2025. They restricted the exercise to objective questions with a verifiable answer. Grammar, writing style, novelty and scientific importance were deliberately excluded.
A first GPT-5 instance searched each PDF for mathematical errors, logical contradictions, miscalculations and disagreements between text, tables and figures. A second GPT-5 instance checked each proposed mistake and removed likely false positives. A smaller model categorised the remaining issues, while another component attempted a correction.
A single confident model critique can itself be wrong. The second pass is intended to reduce hallucinated objections, although it cannot eliminate them. Optical character recognition also created false alarms when equations or indentation were extracted incorrectly.
The Most Common Errors Were Mathematical
Across the full automated audit, 54 per cent of flagged issues involved mathematics or formulas. Textual contradictions accounted for 31.4 per cent, table or figure problems for 9.3 per cent and broken cross-references for 5.3 per cent. The system found examples including an incorrect derivative, a wrong definition and claims that did not match a reported table.
The checker also reported an upward trend. The average number of flags in NeurIPS papers rose from 3.8 in 2021 to 5.9 in 2025. ICLR increased from 4.1 in 2018 to 5.2 in 2025, while the TMLR average moved from 5.0 in 2022 and 2023 to 5.5 in 2025.
Those figures are not proof that modern science is deteriorating. The dataset covers machine learning, publication formats and paper lengths change, and the checker is imperfect. The trend deserves investigation, but it does not establish a universal collapse in research quality.
Human Reviewers Confirmed 263 Mistakes
For the precision test, the authors selected 60 papers in which the checker had reported at least one potentially substantive problem. Human researchers examined all 316 flagged issues and accepted 263 as real mistakes. That produces the reported precision of 83.2 per cent.
The result is strong enough to make the tool useful, but not strong enough to automate accusations. Roughly one in six flags in that sample was false. The selection also focused on papers already flagged as potentially substantive, so it should not be treated as a random human audit of all 2,500 papers.
To test what the system failed to see, the researchers inserted 90 known mistakes into 15 altered versions of five papers. The checker detected 60 per cent overall. It performed best on mathematical errors at 66.7 per cent recall and worst on cross-references at 53.8 per cent. A clean report therefore did not mean a clean paper.
It Could Fix Many Errors, but Not All of Them
Among 240 human-confirmed mistakes considered in the correction experiment, the checker proposed fixes for 207. Human researchers accepted 157 of those proposals, or 75.8 per cent, as correct. In 33 cases the system sensibly returned “No immediate fix” because the contradiction required deeper rewriting or could not be safely resolved from the document alone.
That restraint may be as valuable as the correction rate. A research tool that always produces an answer will disguise uncertainty as productivity. A system that identifies the location of a possible fault and admits that it cannot repair it gives a human expert something bounded to investigate.
The same verification principle appeared when Claude produced a new mathematical lower-bound result. The important step was not accepting an AI-generated proof because it looked persuasive. Human mathematicians examined the argument and a proof assistant supplied an additional formal check.
Another Benchmark Produced a Much Harsher Result
A separate preprint called SPOT tested models against 91 known serious errors taken from 83 papers that had prompted errata or retractions. In its original evaluation, no tested model exceeded 21.1 per cent recall or 6.1 per cent precision. Across eight runs, the systems rarely rediscovered the same errors consistently.
The apparent contradiction is instructive. The Stanford checker searched mostly for local, objective inconsistencies and used multiple model passes. SPOT asked systems to recover serious known failures that can require deeper domain understanding. AI can be useful as a mathematical spell-checker while remaining unreliable as an autonomous scientific referee.
A newer Google system claims substantial progress. The Paper Assistant Tool preprint reports 89.7 per cent recall on its SPOT evaluation after using an agentic pipeline, inference scaling and a logic-aware grader. The authors say the system constructed a counterexample that exposed a fatal gap in one theorem rather than accepting the paper’s claim.
Google Tested Its Reviewer on 4,700 Submissions
Google offered its Paper Assistant Tool to authors preparing submissions for the 2026 STOC and ICML conferences. More than 4,700 manuscripts were processed using an advanced version of Gemini 2.5 Deep Think. The tool was advisory and was not part of the formal acceptance decision.
In surveys reported by the Google-affiliated researchers, 11.6 per cent of STOC respondents and 35.4 per cent of ICML respondents said the tool found substantive theoretical gaps. Thirty-one per cent of ICML respondents said they ran new experiments because of its review. Authors also described bugs and invalid proofs they said had escaped months of work.
These are first-party survey results from a preprint, not an independent audit. Author reports do not prove every AI criticism was correct. The paper itself records hallucinated dates, PDF parsing problems and false claims that valid arguments were wrong.
AI Verification Still Needs Verification
The emerging workflow is a loop rather than a handover. An AI identifies a suspicious equation, a human checks the objection, software may test the correction and the author remains responsible for the final claim. In experimental science, the chain must continue into physical evidence. LiveAIWire’s report on 16 functioning AI-designed bacteriophages mattered because researchers actually built and tested the designs, while most candidates failed.
Automated review also creates new risks. Researchers may learn to write papers that satisfy the checker without improving the underlying science. A widely used reviewer could impose one model’s assumptions across a field, penalise unconventional methods or generate plausible objections that consume more expert time than they save.
The current evidence supports a limited but valuable role. AI is becoming good at finding local, checkable inconsistencies across thousands of pages. It is not yet a dependable judge of whether a paper is important, whether a result will reproduce or whether an unfamiliar idea is genuinely wrong.
The scientific record does not need a machine that declares itself infallible. It needs a tireless assistant that marks where humans should look twice. Across 2,500 published papers, that assistant found hundreds of real reasons to look.
About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, cybersecurity, and the social impact of emerging technology. He publishes daily at LiveAIWire.com.