NatureBench Asks Whether Coding Agents Can Beat Published Science — Mostly, They Can't

A new benchmark is putting the "AI scientist" narrative to its sternest test yet. NatureBench, released by FrontisAI researchers on arXiv, evaluates whether AI coding agents can move beyond reproducing published research toward genuinely advancing it — by challenging them to beat the published state-of-the-art results of peer-reviewed Nature-family papers.

The Setup

NatureBench comprises 90 tasks distilled from Nature-family publications across disciplines. Each task is built through NatureGym, an automated pipeline that constructs a standardized, containerized environment from a source paper — addressing the environment-fragmentation problem that has undermined the credibility of earlier agent-on-research benchmarks. Agents work under a strict web-search-disabled protocol, preventing them from simply looking up the answers they are meant to discover.

The Results

Across ten frontier agent configurations, the strongest model surpassed the published SOTA on only 17.8 percent of tasks under the benchmark's effect-size criterion. That is far from nothing — a sixth of real scientific baselines beaten by an autonomous agent is a result that would have seemed like science fiction three years ago — but it is equally far from the automated-discovery future implied by recent product launches.

The more revealing finding is how agents succeed. Pathway analysis shows wins come primarily through methodological translation: agents convert scientific tasks into familiar supervised prediction problems and apply well-worn machine learning machinery, rather than inventing new scientific approaches. When a task cannot be reframed as a standard ML problem, agents rarely improve on human researchers.

Why It Matters Now

The benchmark lands in a week when AI-for-science ambitions are escalating rapidly: Anthropic has launched Claude Science and its own drug discovery programs, Sakana's AI Scientist has been published in Nature, and coding agents are routinely credited with expert-level software engineering. NatureBench offers the field a calibrated reality check — and a public leaderboard with maintainer-side reproduction to track how fast the gap closes.

The authors have released the benchmark, the NatureGym pipeline and the leaderboard publicly. If the past two years of benchmark history are a guide, 17.8 percent will not survive long — which is precisely what makes the number worth recording.