New Paper: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs

A new arXiv paper offers one of the cleanest demonstrations yet of a suspicion long held by evaluation researchers: frontier language models that ace standard multiple-choice benchmarks can fall apart when the same underlying questions are restructured to require genuine compositional reasoning.

The paper — "From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs," by Hanmeng Liu, Shichao Weng, Xiulai Liu, Zhicai Zhang, Anli Yan and Xiaozhang Liu — targets a structural weakness of the benchmarks that dominate LLM leaderboards: multiple-choice tests age quickly and are vulnerable to data contamination.

From Picking Answers to Judging Combinations

The authors introduce LogiHard, a formal framework that deterministically transforms what they call 0-order selection — picking one answer from a list — into 2-order logical judgment, where a model must evaluate combinations of elements rather than select a single option. The framework incorporates Item Response Theory for adaptive testing with improved difficulty calibration, and the transformation is deterministic: hardened questions are derived mechanically from existing ones rather than written fresh.

Using this machinery, the team built LogiHard-2k, a dataset constructed by analyzing model reasoning patterns across nine dimensions and applying combinatorial transformations to source questions.

The Results

Tested across twelve advanced models, the hardened questions produced accuracy drops of 31-56%. The models also exhibited distinctive failure modes — what the authors describe as multi-select failure and early exit bias — patterns not seen in human performance on the same material.

The most striking result came from transferring the method to MMLU, one of the most widely reported benchmarks in the field: accuracy degraded by 47%, from 89.84% to 42.86%. Because the hardened questions test the same knowledge as the originals, the authors argue the collapse reflects reasoning limitations rather than knowledge gaps. A model that knows a fact but cannot deploy it inside a compositional judgment was, in some sense, never being tested on reasoning at all.

A Crowded Reckoning for Static Benchmarks

The paper lands amid mounting evidence that static benchmarks are losing their evidentiary value. Data contamination — test questions leaking into training corpora — has driven projects like LiveCodeBench to continuously harvest fresh problems that postdate model training cutoffs. Annotation errors are another front: Epoch AI's June revision of FrontierMath corrected flaws found in 42% of its problems. And vendor-run benchmarks, increasingly common in model launches, add a third layer of doubt.

Combinatorial hardening offers a complementary defense. Rather than racing to write new questions faster than they leak, it mechanically regenerates harder variants from existing material — making memorization of the original answer key useless. If the LogiHard results replicate, the uncomfortable implication is that a meaningful share of current leaderboard performance measures familiarity with a question format, not the reasoning the format was meant to probe.