The Inference Report

The SWE-rebench rankings remain stable at the top, with AnthropicFable 5 holding 64.5% and the next five models clustered within 1.5 percentage points, all within their reported confidence intervals. The consistency across both measurement points suggests the gap between frontier code agents has plateaued: Fable 5, Grok 4.5, and Opus 5 occupy positions 1 through 3 with no reordering, and the confidence bands are tight enough that sub-point movements carry little weight. Below rank 10, however, the Artificial Analysis benchmark reveals a different picture. Claude Opus 5 leads that leaderboard at 63.1, beating Claude Fable 5 by a full point despite Fable 5's dominance on SWE-rebench. This divergence suggests the two benchmarks measure different problem classes or evaluation conditions: SWE-rebench may weight integration and repository-level reasoning more heavily, while Artificial Analysis may favor broader reasoning or code understanding. DeepSeek-V4 Pro ranks 14th on SWE-rebench at 40.2% but falls to 30th on Artificial Analysis at 45.3, a reversal that points to distinct test distributions rather than measurement error. The Artificial Analysis list extends to 432 entries, including models scoring 1.0 that likely represent floor effects or incomplete evaluation, whereas SWE-rebench stops at 17 entries with meaningful differentiation. Neither benchmark shows discontinuous jumps or evidence of retesting volatility; the movement is meaningful only insofar as it confirms that code-solving ability measured in controlled repository environments does not correlate perfectly with performance on broader reasoning tasks, and that frontier models have reached a capability ceiling on SWE-rebench that differentiates them from second-tier systems but not from each other.

Cole Brennan