The Inference Report

AnthropicFable 5 maintains its lead on SWE-rebench at 64.5 percent with a confidence interval of plus or minus 1.41 percent, unchanged from the previous measurement, while GrokGrok 4.5 holds second place at 63.8 percent and AnthropicOpus 5 remains third at 63.4 percent, all three separated by margins within their measurement uncertainty. The top tier shows stability: the spread between first and fifth place spans just 2.2 percentage points, with Z.aiGLM-5.2 at 62.9 percent and OpenAIGPT-5.6 Sol at 62.3 percent, both marked as high-capability models. Below this tier, the rankings diverge sharply between the two evaluations, particularly for agent-based systems like JunieJunieAgent (61.8 percent on SWE-rebench, absent from Artificial Analysis) and AnthropicClaude CodeAgent (60.4 percent on SWE-rebench, unranked in Artificial Analysis), suggesting the two benchmarks measure distinct aspects of coding capability or that agent implementations perform better on repository-level tasks than on the broader evaluation set. The Artificial Analysis ranking places Claude Opus 5 first at 60.7 points and Claude Fable 5 second at 59.9 points, reversing their order on SWE-rebench and indicating either different test distributions or that general instruction-following performance does not correlate cleanly with software engineering problem-solving on controlled tasks. The SWE-rebench methodology appears more sensitive to agentic scaffolding and iteration, given agent systems' elevated positions there, while Artificial Analysis weights raw model capability more heavily, as evidenced by reasoning models like o1 and o3 appearing far lower (ranks 121 and 93 respectively) despite their problem-solving reputation. Neither benchmark shows the dramatic shifts that would indicate methodological drift, but the consistent reordering of Anthropic and OpenAI variants across the two systems warrants attention to how each constructs its evaluation tasks, particularly whether SWE-rebench's focus on repository context and tool use versus Artificial Analysis's emphasis on single-turn reasoning creates genuinely different capability profiles or simply reflects different measurement error.

Cole Brennan