Home / Research

What we found checking 65 arXiv papers

Published preprints from eight fields, checked for whether each paper agrees with itself. Almost all of them had something to fix.

DeskReady · Published 16 October 2026

We took 65 preprints posted to arXiv between May 2022 and July 2024, from computer science, economics, experimental particle physics, astrophysics, quantitative biology, electrical engineering, mathematics and experimental nuclear physics. Several of them have since been published at major venues. Each one went through the same check a DeskReady user gets: 1,663 pages in all.

The headline numbers

• 60 of 65 papers had at least one contradiction, number that didn't add up, or figure or table that disagreed with the text.
• Every paper had at least one thing to fix; the median paper had 15.
• 1,488 findings in total: 1,013 to fix and 475 to polish (repeated statements and wording).

What we found, by type

By field

Fields with two to six papers are small samples; read them as examples, not rates.

Why these slips survive a read-through

Of the 171 contradiction, number and figure findings that involve two places in a paper, 73% had their two sides on different pages, and 22% were five or more pages apart: a number in the introduction against a table in the appendix, a caption against the paragraph that discusses its figure three pages later. Reading front to back, you see each place on its own and both look fine.

Longer papers

Long papers did not get proportionally more findings. That says more about the check than about long papers: the version used here reported a similar number of findings per paper whatever its length, so long papers were covered less thoroughly. The current version checks every paragraph, table and figure on its own, which is one of the things the next re-run will show.

How reliable these findings are

A check is only useful if its findings are right. In a separate verification of the run just before this one, 297 findings in 19 of these papers were re-checked independently against the page:

• 92% of all findings were correct; 81% of the ones rated must fix or should fix.
• The weaker categories were the hardest ones: text vs table (62% correct), arithmetic (70%) and figures vs text (78%). Contradictions (93%), references (100%) and wording (99%) were rarely wrong.

Those weak spots are what we have worked on since: numbers in the text are now recomputed against the tables, every figure is read on its own, and every reference is looked up in bibliographic databases. The numbers on this page come from the version of the check before those changes.

Re-run with every release

We will run the same 65 papers again with each new version of DeskReady and update this page, so the numbers show what changed: findings gained, findings lost, and whether the share of correct findings moves. Each update keeps its date at the top.

Examples

A selection of findings that were confirmed against the page in the verification and rated useful to the author. We describe them by field only: the point is how common these slips are, not who made them.

• Quantitative biology, numbers. A dataset section gives 98,221 background and 8,000 positive samples, 106,221 in all; the summary table says 105,263, and the appendix table lists 97,326 and 7,937.
• Quantitative biology, caption vs table. A table caption reports n = 625,229; the main table and the text give 612,081, the sum of the train, validation and test splits.
• Astrophysics, text vs table. An integration time is 515.265 s in the table and 515.365 s in the text.
• Astrophysics, contradiction. The introduction mentions six candidate galaxies from earlier work; the data section and a figure use thirteen, without saying why.
• Electrical engineering, contradiction. The embedding dimension is stated as 768, but the encoder's input and the shared output are given as 128-dimensional.
• Electrical engineering, figure labels. Two panels of one figure are both labelled (c).
• Computer science, claim vs table. The text says one family of methods is more advantageous; the table shows the other family scoring higher on one of the datasets.
• Computer science, naming. A method component has one name throughout the text and a different one in a figure caption and a table caption.
• Economics, structure. Two different propositions in the appendix are both numbered S1.
• Economics, table. Two columns of a supplementary table are swapped between model specifications.
• Particle physics, figure caption. A caption describes uncertainties as horizontal bars; the plot draws them as vertical error bars.
• Particle physics, reference. The citation for a luminosity measurement points to the paper for a different run and energy; the right one is cited elsewhere in the same paper.

Seven more, shown on the papers' own pages with the words highlighted, are on the home page.

Method

• Papers: 65 preprints from eight arXiv archives (cs, econ, hep-ex, astro-ph, q-bio, eess, nucl-ex, math), posted between 2022 and 2024, most in 2023 and 2024; 1,663 pages.
• Check: each PDF as published, through the same check a user gets. Findings are counted as reported; a finding whose quote couldn't be located on the page was dropped before counting.
• Groups: the same as in a DeskReady report. "To fix" covers contradictions and numbers, figures and tables, references, structure and typos; "to polish" covers repeated statements and wording.
• Reliability: a separate verification of 297 findings in 19 of the papers, described above.

Download the data (CSV): one row per paper with its field, year, length range and the number of findings in each group. Paper identifiers are left out, and lengths are given as ranges so rows can't be matched to papers.

What this means for your paper

None of these are signs of bad science. They are the slips that survive careful proofreading because the author knows what the paper is supposed to say: a number updated in the table but not in the text, a caption written for an earlier figure, a claim from an early draft. Reviewers notice them, and they make reviewers trust the rest less. The pre-submission checklist covers how to find them by hand.