The Inference Report

The SWE-rebench leaderboard shows consolidation at the top with no movement among the leading seven models, while mid-tier performers reveal more volatility. Claude Sonnet 4.6 climbed from #10 to maintain position with 51.3 percent, Gemini 3.1 Pro Preview held at #11 with 51.1 percent, and GLM-5.1 remained at #12 with 50.7 percent, though the Artificial Analysis benchmark tells a different story: GLM-5.1 jumped from rank 23 at 40.2 to rank 12 at 50.7, a 10.5-point gain that suggests either a model update or a methodology shift between the two benchmarks. The most striking movement came from GLM-4.7, which advanced from #51 on Artificial Analysis (33.8) to #17 on SWE-rebench (38.2), a 4.4-point improvement, while Kimi K2.6 moved from rank 16 to 15 with a 3.7-point jump from 42.8 to 46.5. These discrepancies between the two benchmarks raise questions about their evaluation methodologies: SWE-rebench appears to reward different model behaviors or architectural choices than Artificial Analysis, particularly for Chinese-developed models like GLM and Kimi, which suggests the benchmarks may be measuring distinct aspects of coding capability rather than converging on a unified signal. The lack of score inflation at the frontier, where the top model remains at 62.7 percent, indicates the evaluation has not become easier, though the divergence between benchmark rankings for identical models undermines confidence in any single leaderboard as a complete measure of coding performance.

Cole Brennan