Source
What We Learned by Reproducing 2,200 papers from ICML
Over 1,200 participants used coding agents in a hackathon to reproduce claims from 2,226 ICML 2026 papers. The project assessed the reproducibility of AI research using automated tools at scale.
Why it matters
As AI publications outpace human review capacity, agent-based auditing can identify scientific errors quickly. This helps researchers and developers avoid relying on flawed theories or incorrect data.
The details
- 51% of examined papers had at least one independently verified claim.
- 23% of papers had at least one claim falsified or contested.
- Human steering remains essential to prevent agents from making scale-dependent errors.
Show entities and relationshipsHide entities and relationships
In this article
Topics
Organizations
Technologies
Companies
Key connections
OpenResearch owns orx
OpenResearch created the orx coding agent.
Hugging Face owns HF Jobs
Hugging Face provides the HF Jobs cloud compute service.
Hugging Face owns Hugging Face Spaces
Hugging Face operates the Hugging Face Spaces hosting platform.
Hugging Face owns Hugging Face Datasets
Hugging Face operates the Hugging Face Datasets repository.
Claude Code competes with Cursor
Both are AI-assisted coding tools used in reproduction experiments.
Both are AI coding agents used by participants.
Show 20 more connectionsShow fewer connections
Claude Code competes with orx
Both are coding agent frameworks used in the hackathon.
Hugging Face is a partner of alphaXiv
Hugging Face and alphaXiv co-organized the ICML 2026 Open Reproductions challenge.
Hugging Face is related to ICML
Hugging Face provided compute credits and hosted reproduction logbooks for ICML 2026 papers.
alphaXiv co-organized the challenge auditing accepted ICML 2026 papers.
Trackio uses Hugging Face Spaces
Trackio reproduction logbooks were published as static Hugging Face Spaces.
Trackio uses Hugging Face Datasets
Full agent execution traces from Trackio logbooks were uploaded to Hugging Face Datasets.
6,816 Trackio logbooks were published attempting to reproduce ICML 2026 papers.
GLM-5.2 served as the automated Logbook Judge assessing reproduction claims for ICML papers.
Claude Code is related to ICML
Claude Code was deployed by participants to reproduce ICML 2026 paper claims.
Codex was used by participants to reproduce ICML 2026 paper claims.
Cursor was used by participants to reproduce ICML 2026 paper claims.
Pi was used by participants to reproduce ICML 2026 paper claims.
orx was used by participants to reproduce ICML 2026 paper claims.
HF Jobs cloud instances were launched to run experiments replicating ICML papers.
Authors uploaded revised versions of ICML papers to arXiv following reproduction findings.
Claude Code uses Coding Agents
Claude Code operates as an AI coding agent.
GLM-5.2 uses LLM Judges
GLM-5.2 was utilized as an automated LLM judge to evaluate reproduction claims.
Hugging Face is related to Machine Learning
Hugging Face builds infrastructure and tools for machine learning.
ICML is related to Machine Learning
ICML is an academic conference focused on machine learning research.
Claude Code competes with Codex
Claude Code and Codex compete for share of coding agent traffic on the Hub.
Related events
ICML 2026 Open Reproductions Challenge
Get the weekly recap
The stories like this one, picked and explained — once a week, straight to your inbox.