The Inference Report

On the SWE-rebench, the top tier has stabilized with Claude Code holding 52.9%, Junie at 52.1%, and Claude Opus 4.6 tied with gpt-5.2-xhigh at 51.7%, representing no movement from prior rankings. Claude Opus 4.5 dropped from position 8 at 49.7% on Artificial Analysis to position 12 at 43.8% on SWE-rebench, a 5.9-point decline that signals either methodological differences between the two benchmarks or genuine performance variance across problem distributions. Kimi K2 Thinking climbed 14 positions on Artificial Analysis (from 27 to 13) with a 2.9-point gain, while Gemini 3 Pro Preview fell from position 10 to 11 despite holding steady at 48.4%, indicating a new entrant shifted rankings. GLM-5 dropped significantly on Artificial Analysis from position 7 at 49.8% to position 15 at 42.1%, a 7.7-point regression that merits scrutiny regarding whether this reflects model degradation or evaluation instability. Kimi K2.5 declined sharply from position 12 at 46.8% to position 19 at 37.9%, losing 8.9 points and 7 ranking positions. The Artificial Analysis leaderboard saw new entries at position 10 (Grok 4.20 Beta) and position 40 (NVIDIA Nemotron 3 Super 120B), while position 97 (LongCat Flash Lite) and position 282 (Sarvam M) entered lower tiers, suggesting either benchmark expansion or periodic model rotation. The divergence between SWE-rebench and Artificial Analysis on models like Claude Opus 4.5 and GLM-5 raises questions about benchmark sensitivity to implementation details or task sampling; without clarity on evaluation methodology differences, it is difficult to assess whether these gaps reflect real capability variation or measurement artifacts.

Cole Brennan