FrontierMath v2: Epoch's Error-Corrected Gauntlet for AI Mathematicians

Epoch AI released FrontierMath v2 on June 12, and buried in the announcement was a startling admission: an audit had found errors in 42% of the original benchmark's problems. For the evaluation suite most often cited as the gold standard for frontier mathematical reasoning, that number is both an embarrassment and a model of how benchmark maintenance should work.

What Got Fixed

The v2 dataset now comprises 338 problems: a base set of 295 spanning Tiers 1 through 3, plus an expansion set of 43 exceptionally difficult Tier 4 problems pitched at research-level mathematics. According to Epoch's benchmark hub, the revision corrected 123 problems in Tiers 1-3 and 12 in Tier 4, while removing 5 and 7 problems respectively that could not be salvaged.

The problems were crafted and vetted by expert mathematicians and span most major branches of modern mathematics — from computationally intensive number theory and real analysis to abstract questions in algebraic geometry and category theory. Typical problems demand multiple hours from a working researcher; upper-tier questions can take days. The benchmark was built with funding from OpenAI, which retains exclusive access to a subset of problems — an arrangement Epoch discloses in a published conflict-of-interest statement.

Why Error Correction Matters

The most telling detail, highlighted in analysis by Digital Applied, is the direction of the errors: many were marking correct answers as wrong. When the fixes landed, model scores rose while rankings remained broadly stable — evidence that the flaws had been suppressing measured capability rather than inflating it.

That cuts against a common assumption in benchmark skepticism, which usually worries about contamination pushing scores up. Here the opposite failure mode was in play: annotation error was hiding genuine progress. For a benchmark where problems are designed so that guessing succeeds less than 1% of the time, a mislabeled answer key is not noise — it is a hard ceiling on the score any model, however capable, can achieve.

The episode suggests that as frontier models close in on expert-level performance, benchmark error rates become the binding constraint on measurement. A 42% problem-level flaw rate would barely matter when models score 2%, as they did on FrontierMath's hardest material in late 2024. It matters enormously once they are contending for the top of the scale.

The Moving Target

FrontierMath now occupies an unusual position: it is simultaneously one of the last benchmarks frontier models have not saturated and a live demonstration that "unsaturated" partly depended on grading mistakes. Epoch's willingness to audit, correct, and re-baseline its own flagship — publicly, with the removals itemized — sets a standard the rest of the evaluation ecosystem has been slow to adopt.

Whether the 43 Tier 4 problems hold out against the next generation of reasoning models is an open question. What v2 establishes is narrower but important: before asking whether AI can do research mathematics, someone has to verify the answer key.