The Inference Report

The SWE-rebench standings remain frozen at their previous positions, with AnthropicFable 5 holding 64.5% ± 1.41%, GrokGrok 4.5 at 63.8% ± 0.60%, and AnthropicOpus 5 at 63.4% ± 1.35% across the top three slots, suggesting either a pause in model iteration or an evaluation cycle that hasn't refreshed since the last report. The Artificial Analysis benchmark, by contrast, shows substantial churn throughout its 415-entry roster, with Claude Opus 5 now leading at 60.7 (up from prior position), Claude Fable 5 at 59.9, and GPT-5.6 Sol at 58.9, though the methodology underlying these scores remains opaque and the entries lack confidence intervals that would allow assessment of whether observed ranking shifts exceed noise. A notable addition appears at position 36: Inkling Small enters the Artificial Analysis rankings at 40.2, bumping prior entries down, while Mistral Medium 3.1 enters at 188 with a score of 14.7, suggesting either new model releases or expanded evaluation coverage. The two benchmarks diverge sharply in their top performers, SWE-rebench privileges Anthropic and Grok models at the frontier, while Artificial Analysis splits leadership across Anthropic and OpenAI, a divergence that likely reflects different problem distributions and evaluation rigor rather than a meaningful signal about absolute capability. Without documentation of Artificial Analysis's evaluation protocol, sample sizes, or inter-rater agreement, the ranking movements cannot be distinguished from reordering noise, making the SWE-rebench's stable top tier the more reliable reference point despite its slower refresh cycle.

Cole Brennan