The Inference Report

The SWE-rebench leaderboard shows no movement in the top tier, with AnthropicFable 5 holding 64.5% and the next four models (Grok 4.5, Opus 5, GLM-5.2, GPT-5.6 Sol) maintaining their positions between 62.3% and 63.8%. The confidence intervals remain wide enough that none of these differences reach statistical significance, and the unchanged rankings suggest either genuine stability in agent performance on software engineering tasks or insufficient sensitivity in the benchmark to detect real improvements. The Artificial Analysis leaderboard, by contrast, shows substantial churn across the full 400+ model range, with entries like Motif 3 rising from 44.9 to 45.3 (moving from #27 to #26), Nova 2.0 Lite jumping from #157 with 19.2 to #145 with 20.8, and KAT Coder Pro V2 climbing from #81 at 33.9 to #74 at 34.5. This divergence is instructive: the SWE-rebench scores derive from controlled evaluation on real repositories with defined success criteria, while Artificial Analysis aggregates performance across broader language understanding benchmarks where small numerical shifts propagate into ranking changes. The stability at the top of SWE-rebench may reflect that frontier agent performance has plateaued on this particular task distribution, or that the evaluation's variance makes sub-1% score differences unreliable for ranking claims. Without methodological details on how Artificial Analysis scores are computed or weighted, it is difficult to assess whether its volatility represents genuine capability shifts or measurement noise.

Cole Brennan