The Inference Report

The SWE-rebench leaderboard shows no movement across the top tier, with AnthropicFable 5 maintaining 64.5±1.41%, GrokGrok 4.5 at 63.8±0.60%, and AnthropicOpus 5 at 63.4±1.35%, their confidence intervals overlapping sufficiently that ranking stability reflects measurement precision rather than performance divergence. The Artificial Analysis benchmark, by contrast, exhibits substantial churn below the top 100 entries, though the methodology underlying these scores remains opaque compared to SWE-rebench's transparent evaluation against real software engineering tasks. Claude Opus 5 and Claude Fable 5 swap positions at the top of Artificial Analysis (63.1 and 62.1 respectively), while G9v3-39A5B climbs from position 102 to 92, gaining 2.4 points in the process, and KAT Coder Pro V2, Kimi K2 Thinking, and o3-pro each shift up one rank. The mid-tier reshuffling involves models like QwQ 32B and Qwen3 VL 30B A3B trading positions 224 and 225, suggesting either evaluation variance or incremental improvements below the noise floor. SWE-rebench's stability across all 17 ranked models implies the coding task distribution has reached a plateau where current approaches plateau, while Artificial Analysis's frequent reordering raises questions about whether the benchmark captures genuine capability differences or reflects sensitivity to prompt variation and evaluation artifact. Without documented methodology for Artificial Analysis, the practical significance of mid-list movements remains ambiguous.

Cole Brennan