Reproducing 2,226 ICML 2026 Papers with Agents - 23% Had a Claim Falsified, 242 Split the Verdict

On August 13, 2026, Hugging Face published the results of an effort to reproduce, claim by claim, the papers accepted at the machine learning conference ICML 2026, using coding agents that community members brought themselves¹. Over the 19 days from July 15 to August 2, more than 1,200 participants published 6,816 logbooks and attempted 2,226 papers — roughly a third of the conference¹.

Aggregated per paper, 51% of the papers examined (1,103) had at least one claim independently verified, while 23% (496) had at least one claim falsified or contested¹. And on 242 papers, independent reproduction teams reached opposite verdicts on the same claims¹. The post puts it this way: “Reproducibility is not binary; it is adversarial”¹.

What is covered here is one conference, and only 34% of its accepted papers — not a figure representing the reproducibility of machine learning research as a whole¹. What follows assumes that scope.

Review Capacity That Has Not Kept Up

The motivation Hugging Face gives is the gap between submission growth and review capacity. ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year, continuing what it describes as an exponential trend¹. One factor it names is AI agents making it faster to run experiments and write them up¹.

Review capacity has not doubled alongside it. Noting that reviewers at most conferences are volunteers who may lack the time or expertise to fully review a paper, the post quotes a reviewer of an accepted spotlight paper in their own words: “My low confidence score is because I did not check all the proofs carefully”¹. That paper received strong scores and a spotlight¹.

Against that, the post sets out the view that the same technology producing the flood of submissions can also help keep up with it. Checking a paper carefully used to cost a reviewer a weekend; an agent can attempt it in an afternoon, in parallel, thousands of times over¹.

How the Judging Worked

How to read the results depends on how they were judged. Hugging Face describes the mechanism as follows¹.

It indexed 6,341 accepted papers with their abstracts and extracted the core scientific claims of each, so an agent could start from a concrete, checkable target rather than a 40-page PDF; multiple people reproducing the same paper was encouraged¹. The agents participants used varied — Claude Code, Codex, Cursor, OpenResearch’s orx, and others — with no standardization¹.

Each run became a public record in the form of a Trackio logbook, containing the write-up, the code that ran, the artifacts produced, and optionally the full agent execution trace. The principle stated is that the auditing process itself had to be auditable¹.

Judging was handled by an automated Logbook Judge running the open-weights model GLM-5.2¹. That model comes from Z.ai; this site covered the release of its successor, GLM-5.3 on August 15. The judge re-read every logbook and issued a per-claim verdict of verified, falsified, toy (evidence at reduced scale), or inconclusive, and was explicitly instructed to treat each logbook’s self-assessment as untrusted¹.

Participants received $20 in compute credits for HF Jobs, and 2,962 cloud jobs were launched over the period. Where full reproduction was impossible — a proprietary dataset, unreleased checkpoints — reduced-scale runs used synthetic data mimicking the original’s properties¹. In the end 35,908 claims were judged, with all verdicts frozen in a public dataset at the challenge’s close¹.

What 51% and 23% Actually Mean

The definitions need care. The 51% (1,103 papers) is the share with at least one claim independently verified — not papers reproduced in full¹. Papers where every extracted claim was verified, and so fully reproduced, number 266; a further 632 were partially reproduced with nothing falsified. Individual claims confirmed with real experiments total 3,978¹.

Likewise the 23% (496 papers) is the share with at least one claim falsified or contested, of which 49 had all claims falsified and nothing verified¹. The rest sit in between: 502 papers with toy-scale evidence only, and 280 where nothing could be established either way, most often because artifacts were missing¹.

Well-reproduced examples are cited too. “Flat Minima and Generalization: Insights from Stochastic Convex Optimization” was reproduced by 20 independent teams, 12 of which verified every claim¹. “A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness” had 14 of 17 logbooks verify every claim¹.

What Broke

Thirty-five participants formally claimed a falsification, and Hugging Face says it adversarially re-verified all of them — re-reading the paper, re-reading the logbook, and re-deriving the math or re-implementing the experiment from the paper’s own text¹. Among the confirmed cases:

On the spotlight paper cited at the top, “Towards Optimal Robustness in Learning-Augmented Paging,” a participant’s logbook measured the additive term growing like 0.38 ln k and located the exact step where the proof breaks. Hugging Face’s own re-implementation extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma¹.

On “Attention’s forward pass and Frank-Wolfe,” three independent teams found counterexamples. Violations first appeared at t = 224, ~3,800, and 6,416 steps, which is said to explain why other participants returned a verdict of “verified”: finite-horizon checks stop too early¹. The authors confirmed the same day and are working on a fix¹.

On “Self-Distillation Enables Continual Learning,” the paper’s central equation and entire theory section analyze reverse KL divergence, while the released code’s default computes forward KL — the code that, per the authors, produced all of the paper’s results¹. The logbook that caught it also failed to reproduce the paper’s headline +4pp result under the authors’ own code and data¹.

The clearest example of a number changing shape is “Do Transformers Need Three Projections?” Roughly 66% of evaluated label positions were EOS padding tokens that train to near-zero loss, deflating perplexity by about threefold. The abstract’s “3.1% quality cost for 50% cache reduction” becomes roughly 9.4% once corrected¹.

Sometimes the Falsification Is the Thing That’s Wrong

Cases where the falsification itself was mistaken are also published. One logbook claimed a paper’s method was 2x slower than the baseline; this turned out to be an arithmetic bug on the reproduction side, comparing per-trajectory time against per-batch-of-50 time. Correctly normalized, the participant’s own data confirmed the paper’s claimed 8x speedup¹.

Hugging Face has begun writing to authors about confirmed findings, framing it as: here is what we found, here is all the evidence, do you agree or is our analysis wrong¹. So far authors have confirmed findings on multiple papers, two arXiv corrections are in flight, and in one case an author had quietly fixed the error in a new version a month before the challenge found it¹.

Where Agents Alone Did Not Stop

The conclusion drawn is not that agents replaced human review. Hugging Face says agent-only execution hit real limits, citing agents getting stuck in local loops, misreading scale-dependent behavior, and building an entire falsification on top of a units mismatch¹. Multiple “verified” verdicts on the paging paper came from checks that stopped before the log-k growth became visible¹.

The most reliable results, it says, came from workflows where a human was steering: re-pointing the agent, questioning an assumption, or deciding an experiment’s premise was wrong before burning a week of compute¹. In an example involving image generation under extreme quantization, numerical metrics said “no collapse,” but whether the images were usable was a perceptual question — a human personally judged all 128 image pairs, with an agent validating the consistency of those annotations afterward¹.

On the human reviewer’s role, the post says it is “to manage intelligence effectively,” drawing an analogy to a principal investigator setting up an environment where graduate students can do good work. Those who got the most from their agents were the ones who built the right environment and asked the right questions before letting them run¹.

Using Paper Numbers as Evidence

If Hugging Face’s analysis of Hub data on open models, published the same month, dealt with the gap between attention and actual use, this report deals with the gap between claim and verification. Both show, within a bounded population, that the number you see first and what sits behind it are some distance apart.

When choosing models or methods, the numbers on hand are often ones a vendor measured itself. As with the Gemini 3.7 Flash announcement, benchmarks compared only against a vendor’s own prior generation, with no independent verification, are common. What this result adds is that on the peer-reviewed side too, some share of claims either collapses under independent reproduction or splits the verdict. The 242 papers with opposite verdicts mean that a single reproduction does not always settle the matter.

That said, this result cannot be taken unconditionally either. The verdicts came from an automated judge; the 35 falsification claims were re-verified, but not every verified claim went through the same procedure. Participants were volunteers, and neither their agent setups nor their compute budgets were uniform — as the post itself states¹. Even so, because every logbook, verdict, trace, and artifact is public, the merits of any individual judgment can be traced by the reader¹. That verifiability may be the part that is more usable in practice than the headline numbers.