New SWE-Together Benchmark Stress-Tests Multi-Turn Coding Agents

A new benchmark called SWE-Together is pushing coding-agent evaluation past the single-shot paradigm that has dominated the field, testing how agents perform when users clarify goals, add constraints, and correct mistakes across multiple turns.

Reconstructed from real user-agent sessions, SWE-Together curates 109 repository-level tasks drawn from more than 11,000 recorded interactions. To make those sessions replayable across different models, the researchers built a reactive, LLM-based user simulator that preserves each original user's intent and supplies feedback when an agent's progress calls for it. Crucially, the benchmark scores not just final repository correctness but the number of corrective feedback turns an agent needs — a proxy for how much "steering" a human has to do.

On the leaderboard, Claude Opus 4.8 leads with roughly 63 percent pass@1, ahead of GPT-5.5 and Claude Opus 4.6 (both around 58 percent) and GLM-5.2. It also required the fewest corrective interventions, at about 1.38 user-correction turns per trial. The paper reports a strong inverse relationship between capability and hand-holding — stronger agents reach the top of the board while asking less of their users.

The shift matters because real coding assistance is interactive, not one-and-done. As the field moves beyond static SWE-bench-style tasks toward agentic, multi-turn evaluation, SWE-Together offers a sharper picture of which models are genuinely good collaborators.