SciCodePile: A 128GB Corpus and a Brutal Benchmark Where Top Models Solve 12% of Scientific Code Tasks

Amid a summer of coding-agent triumphalism — SWE-bench records, terminal benchmarks in the high 80s, Kimi K3 topping the Frontend Code Arena — a new paper offers a bracing corrective. On SciCodePile, an executable benchmark built from real scientific software, the best of 15 evaluated LLMs passes just 12.30% of tasks on the first attempt.

What SciCodePile is

Introduced in a paper trending on arXiv this week, SciCodePile pairs two things the field has lacked together: a 128GB training-scale corpus of scientific code drawn from 37,737 research repositories across disciplines, and an executable benchmark of 200 tasks wrapped in sandboxed environments with automated test harnesses.

The "executable" part is what gives the result its teeth. Rather than scoring generated code for surface similarity to a reference — the flattering metric many benchmarks quietly rely on — SciCodePile runs it. A solution counts only if it produces the correct output when actually executed. That single design choice tends to collapse the impressive-looking numbers models post elsewhere.

Why scientific code is different

Scientific code is a distinct and unusually hostile domain for language models. It fuses domain expertise (the physics or chemistry the code encodes), numerical correctness (where a subtly wrong constant or unit silently corrupts results), sprawling and idiosyncratic dependencies, and problem statements that assume knowledge no amount of GitHub pretraining reliably supplies. General web-scale code data — heavy on web apps and standard software patterns — underrepresents exactly this territory.

The 12.30% Pass@1 figure quantifies the shortfall: models fluent enough to top web-development leaderboards stumble badly when the task is implementing a research method whose correctness is defined by executable numerical output rather than plausible-looking syntax.

The relevance to Asia's research ambitions

The result matters beyond benchmark-watching. Governments across Asia — Japan's "AI for Science" push, China's high-quality-dataset programs, Korea's and Singapore's research-compute investments — are betting heavily on AI as an accelerant for scientific discovery. SciCodePile is a reminder that the reasoning-and-agents progress dominating the headlines does not automatically transfer to the code that actually runs experiments and simulations. Foundation models for science remain meaningfully behind their general-purpose cousins.

The honest read

The value of a benchmark like SciCodePile is precisely that it resists gaming: you cannot pattern-match your way to a passing execution. Expect the number to climb as labs fine-tune on the accompanying corpus — but expect it to climb slowly, and to reveal, as it does, which models actually understand scientific computing versus which have merely memorized its surface. For anyone deploying coding agents in a research or engineering context, 12.30% is the figure to keep next to the leaderboard screenshots.